2024-03-10T00:37:12.809 INFO:root:teuthology version: 0.0.1.dev262+ge691533 2024-03-10T00:37:12.810 DEBUG:teuthology.run:Teuthology command: teuthology --name yuriw-2024-03-08_16:20:46-rados-wip-yuri4-testing-2024-03-05-0854-distro-default-smithi --owner scheduled_yuriw@teuthology --verbose --archive /home/teuthworker/archive/yuriw-2024-03-08_16:20:46-rados-wip-yuri4-testing-2024-03-05-0854-distro-default-smithi/7587708 --description rados/cephadm/smoke/{0-distro/centos_9.stream_runc 0-nvme-loop agent/on fixed-2 mon_election/connectivity start} -- /home/teuthworker/archive/yuriw-2024-03-08_16:20:46-rados-wip-yuri4-testing-2024-03-05-0854-distro-default-smithi/7587708/orig.config.yaml 2024-03-10T00:37:12.840 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-03-10T00:37:12.929 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/yuriw-2024-03-08_16:20:46-rados-wip-yuri4-testing-2024-03-05-0854-distro-default-smithi/7587708 branch: wip-yuri4-testing-2024-03-05-0854 description: rados/cephadm/smoke/{0-distro/centos_9.stream_runc 0-nvme-loop agent/on fixed-2 mon_election/connectivity start} email: yweinste@redhat.com first_in_suite: false job_id: '7587708' kernel: kdb: true sha1: distro last_in_suite: false machine_type: smithi name: yuriw-2024-03-08_16:20:46-rados-wip-yuri4-testing-2024-03-05-0854-distro-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 4 size: 10 os_type: centos os_version: 9.stream overrides: admin_socket: branch: wip-yuri4-testing-2024-03-05-0854 ceph: conf: global: mon election default strategy: 3 mgr: debug mgr: 20 debug ms: 1 mgr/cephadm/use_agent: true mon: debug mon: 20 debug ms: 1 debug paxos: 20 osd: debug ms: 1 debug osd: 20 osd shutdown pgref assert: true flavor: default log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - MON_DOWN sha1: 831bc188fa69f143bbba69ac76119e784500e8e3 ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} install: ceph: flavor: default sha1: 831bc188fa69f143bbba69ac76119e784500e8e3 selinux: whitelist: - scontext=system_u:system_r:logrotate_t:s0 workunit: branch: wip-yuri4-testing-2024-03-05-0854 sha1: 831bc188fa69f143bbba69ac76119e784500e8e3 owner: scheduled_yuriw@teuthology priority: 71 repo: https://github.com/ceph/ceph-ci.git roles: - - mon.a - mon.c - mgr.y - osd.0 - osd.1 - osd.2 - osd.3 - client.0 - ceph.rgw.foo.a - node-exporter.a - alertmanager.a - - mon.b - mgr.x - osd.4 - osd.5 - osd.6 - osd.7 - client.1 - prometheus.a - grafana.a - node-exporter.b - ceph.iscsi.iscsi.a seed: 6371 sha1: 831bc188fa69f143bbba69ac76119e784500e8e3 sleep_before_teardown: 0 subset: 111/120000 suite: rados suite_branch: wip-yuri4-testing-2024-03-05-0854 suite_path: /home/teuthworker/src/github.com_ceph_ceph-c_831bc188fa69f143bbba69ac76119e784500e8e3/qa suite_relpath: qa suite_repo: https://github.com/ceph/ceph-ci.git suite_sha1: 831bc188fa69f143bbba69ac76119e784500e8e3 targets: smithi106.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBET97ZX/gb8FSJnFNCUlMEvsQi2MTZvHo2itkZIAxRVC2oVnUb1UXZEjIzujvNU7ZF7j6gytQtN6TYgpvmNbdgQ= smithi170.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBGf5wFffxIrQcRI0K0bQr3WXg0F9WC1OE706cGuypPKp03l1oIpUATODTmpzKJAzWPPUXVcdGCWKDUvu+K+5h8o= tasks: - pexec: all: - sudo dnf install runc -y - sudo sed -i 's/^#runtime = "crun"/runtime = "runc"/g' /usr/share/containers/containers.conf - sudo sed -i 's/runtime = "crun"/#runtime = "crun"/g' /usr/share/containers/containers.conf - nvme_loop: null - cephadm: conf: mgr: debug mgr: 20 debug ms: 1 - cephadm.shell: mon.a: - stat -c '%u %g' /var/log/ceph | grep '167 167' - ceph orch status - ceph orch ps - ceph orch ls - ceph orch host ls - ceph orch device ls - ceph orch ls --format yaml - ceph orch ls | grep '^osd ' teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: e691533f9cbb33d85b2187bba20d7102f098636d timestamp: 2024-03-08_16:20:46 tube: smithi user: yuriw verbose: true worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 2024-03-10T00:37:12.929 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_ceph_ceph-c_831bc188fa69f143bbba69ac76119e784500e8e3/qa; will attempt to use it 2024-03-10T00:37:12.930 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_ceph_ceph-c_831bc188fa69f143bbba69ac76119e784500e8e3/qa/tasks 2024-03-10T00:37:12.930 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-03-10T00:37:12.931 INFO:teuthology.task.internal:Checking packages... 2024-03-10T00:37:12.956 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash '831bc188fa69f143bbba69ac76119e784500e8e3' 2024-03-10T00:37:12.956 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-03-10T00:37:12.956 INFO:teuthology.packaging:ref: None 2024-03-10T00:37:12.956 INFO:teuthology.packaging:tag: None 2024-03-10T00:37:12.956 INFO:teuthology.packaging:branch: wip-yuri4-testing-2024-03-05-0854 2024-03-10T00:37:12.956 INFO:teuthology.packaging:sha1: 831bc188fa69f143bbba69ac76119e784500e8e3 2024-03-10T00:37:12.956 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=wip-yuri4-testing-2024-03-05-0854 2024-03-10T00:37:13.219 INFO:teuthology.task.internal:Found packages for ceph version 19.0.0-1875.g831bc188 2024-03-10T00:37:13.220 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-03-10T00:37:13.227 INFO:teuthology.task.internal:no buildpackages task found 2024-03-10T00:37:13.227 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-03-10T00:37:13.242 INFO:teuthology.task.internal:Saving configuration 2024-03-10T00:37:13.253 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-03-10T00:37:13.259 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-10T00:37:13.284 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi106.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2024-03-08_16:20:46-rados-wip-yuri4-testing-2024-03-05-0854-distro-default-smithi/7587708', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-10 00:32:12.960846', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBET97ZX/gb8FSJnFNCUlMEvsQi2MTZvHo2itkZIAxRVC2oVnUb1UXZEjIzujvNU7ZF7j6gytQtN6TYgpvmNbdgQ='} 2024-03-10T00:37:13.300 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi170.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2024-03-08_16:20:46-rados-wip-yuri4-testing-2024-03-05-0854-distro-default-smithi/7587708', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-10 00:32:12.958913', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBGf5wFffxIrQcRI0K0bQr3WXg0F9WC1OE706cGuypPKp03l1oIpUATODTmpzKJAzWPPUXVcdGCWKDUvu+K+5h8o='} 2024-03-10T00:37:13.301 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-03-10T00:37:13.306 INFO:teuthology.task.internal:roles: ubuntu@smithi106.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'ceph.rgw.foo.a', 'node-exporter.a', 'alertmanager.a'] 2024-03-10T00:37:13.307 INFO:teuthology.task.internal:roles: ubuntu@smithi170.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'grafana.a', 'node-exporter.b', 'ceph.iscsi.iscsi.a'] 2024-03-10T00:37:13.307 INFO:teuthology.run_tasks:Running task console_log... 2024-03-10T00:37:13.373 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f722ace4160>, signals=[15]) 2024-03-10T00:37:13.374 INFO:teuthology.run_tasks:Running task internal.connect... 2024-03-10T00:37:13.380 INFO:teuthology.task.internal:Opening connections... 2024-03-10T00:37:13.380 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi106.front.sepia.ceph.com 2024-03-10T00:37:13.382 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi106.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-10T00:37:13.480 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi170.front.sepia.ceph.com 2024-03-10T00:37:13.481 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi170.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-10T00:37:13.579 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-03-10T00:37:13.586 DEBUG:teuthology.orchestra.run.smithi106:> uname -m 2024-03-10T00:37:13.604 INFO:teuthology.orchestra.run.smithi106.stdout:x86_64 2024-03-10T00:37:13.604 DEBUG:teuthology.orchestra.run.smithi106:> cat /etc/os-release 2024-03-10T00:37:13.660 INFO:teuthology.orchestra.run.smithi106.stdout:NAME="CentOS Stream" 2024-03-10T00:37:13.661 INFO:teuthology.orchestra.run.smithi106.stdout:VERSION="9" 2024-03-10T00:37:13.661 INFO:teuthology.orchestra.run.smithi106.stdout:ID="centos" 2024-03-10T00:37:13.661 INFO:teuthology.orchestra.run.smithi106.stdout:ID_LIKE="rhel fedora" 2024-03-10T00:37:13.661 INFO:teuthology.orchestra.run.smithi106.stdout:VERSION_ID="9" 2024-03-10T00:37:13.661 INFO:teuthology.orchestra.run.smithi106.stdout:PLATFORM_ID="platform:el9" 2024-03-10T00:37:13.661 INFO:teuthology.orchestra.run.smithi106.stdout:PRETTY_NAME="CentOS Stream 9" 2024-03-10T00:37:13.661 INFO:teuthology.orchestra.run.smithi106.stdout:ANSI_COLOR="0;31" 2024-03-10T00:37:13.661 INFO:teuthology.orchestra.run.smithi106.stdout:LOGO="fedora-logo-icon" 2024-03-10T00:37:13.661 INFO:teuthology.orchestra.run.smithi106.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-03-10T00:37:13.661 INFO:teuthology.orchestra.run.smithi106.stdout:HOME_URL="https://centos.org/" 2024-03-10T00:37:13.661 INFO:teuthology.orchestra.run.smithi106.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-03-10T00:37:13.661 INFO:teuthology.orchestra.run.smithi106.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-03-10T00:37:13.661 INFO:teuthology.orchestra.run.smithi106.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-03-10T00:37:13.662 INFO:teuthology.lock.ops:Updating smithi106.front.sepia.ceph.com on lock server 2024-03-10T00:37:13.682 DEBUG:teuthology.orchestra.run.smithi170:> uname -m 2024-03-10T00:37:13.699 INFO:teuthology.orchestra.run.smithi170.stdout:x86_64 2024-03-10T00:37:13.700 DEBUG:teuthology.orchestra.run.smithi170:> cat /etc/os-release 2024-03-10T00:37:13.757 INFO:teuthology.orchestra.run.smithi170.stdout:NAME="CentOS Stream" 2024-03-10T00:37:13.757 INFO:teuthology.orchestra.run.smithi170.stdout:VERSION="9" 2024-03-10T00:37:13.757 INFO:teuthology.orchestra.run.smithi170.stdout:ID="centos" 2024-03-10T00:37:13.757 INFO:teuthology.orchestra.run.smithi170.stdout:ID_LIKE="rhel fedora" 2024-03-10T00:37:13.757 INFO:teuthology.orchestra.run.smithi170.stdout:VERSION_ID="9" 2024-03-10T00:37:13.757 INFO:teuthology.orchestra.run.smithi170.stdout:PLATFORM_ID="platform:el9" 2024-03-10T00:37:13.757 INFO:teuthology.orchestra.run.smithi170.stdout:PRETTY_NAME="CentOS Stream 9" 2024-03-10T00:37:13.757 INFO:teuthology.orchestra.run.smithi170.stdout:ANSI_COLOR="0;31" 2024-03-10T00:37:13.757 INFO:teuthology.orchestra.run.smithi170.stdout:LOGO="fedora-logo-icon" 2024-03-10T00:37:13.757 INFO:teuthology.orchestra.run.smithi170.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-03-10T00:37:13.757 INFO:teuthology.orchestra.run.smithi170.stdout:HOME_URL="https://centos.org/" 2024-03-10T00:37:13.757 INFO:teuthology.orchestra.run.smithi170.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-03-10T00:37:13.758 INFO:teuthology.orchestra.run.smithi170.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-03-10T00:37:13.758 INFO:teuthology.orchestra.run.smithi170.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-03-10T00:37:13.758 INFO:teuthology.lock.ops:Updating smithi170.front.sepia.ceph.com on lock server 2024-03-10T00:37:13.782 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-03-10T00:37:13.795 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-03-10T00:37:13.801 INFO:teuthology.task.internal:Checking for old test directory... 2024-03-10T00:37:13.801 DEBUG:teuthology.orchestra.run.smithi106:> test '!' -e /home/ubuntu/cephtest 2024-03-10T00:37:13.803 DEBUG:teuthology.orchestra.run.smithi170:> test '!' -e /home/ubuntu/cephtest 2024-03-10T00:37:13.820 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-03-10T00:37:13.826 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-03-10T00:37:13.826 DEBUG:teuthology.orchestra.run.smithi106:> test -z $(ls -A /var/lib/ceph) 2024-03-10T00:37:13.860 DEBUG:teuthology.orchestra.run.smithi170:> test -z $(ls -A /var/lib/ceph) 2024-03-10T00:37:13.888 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-03-10T00:37:13.943 INFO:teuthology.run_tasks:Running task kernel... 2024-03-10T00:37:13.959 INFO:teuthology.task.kernel:normalize config orig: {'kdb': True, 'sha1': 'distro'} 2024-03-10T00:37:13.959 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': True, 'sha1': 'distro'}, 'mon.b': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2024-03-10T00:37:13.959 DEBUG:teuthology.orchestra.run.smithi106:> test -f /run/.containerenv -o -f /.dockerenv 2024-03-10T00:37:13.959 DEBUG:teuthology.orchestra.run.smithi170:> test -f /run/.containerenv -o -f /.dockerenv 2024-03-10T00:37:13.975 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-10T00:37:13.976 DEBUG:teuthology.orchestra.run.smithi106:> uname -r 2024-03-10T00:37:13.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-10T00:37:13.977 DEBUG:teuthology.orchestra.run.smithi170:> uname -r 2024-03-10T00:37:14.033 INFO:teuthology.orchestra.run.smithi106.stdout:5.14.0-361.el9.x86_64 2024-03-10T00:37:14.033 INFO:teuthology.task.kernel:Running kernel on smithi106: 5.14.0-361.el9.x86_64 2024-03-10T00:37:14.034 DEBUG:teuthology.orchestra.run.smithi106:> sudo yum install -y kernel 2024-03-10T00:37:14.035 INFO:teuthology.orchestra.run.smithi170.stdout:5.14.0-361.el9.x86_64 2024-03-10T00:37:14.035 INFO:teuthology.task.kernel:Running kernel on smithi170: 5.14.0-361.el9.x86_64 2024-03-10T00:37:14.035 DEBUG:teuthology.orchestra.run.smithi170:> sudo yum install -y kernel 2024-03-10T00:37:16.018 INFO:teuthology.orchestra.run.smithi170.stdout:CentOS-9 - AppStream 15 MB/s | 6.7 MB 00:00 2024-03-10T00:37:16.146 INFO:teuthology.orchestra.run.smithi106.stdout:CentOS-9 - AppStream 14 MB/s | 6.7 MB 00:00 2024-03-10T00:37:17.927 INFO:teuthology.orchestra.run.smithi170.stdout:CentOS-9 - BaseOS 2.5 MB/s | 2.0 MB 00:00 2024-03-10T00:37:17.972 INFO:teuthology.orchestra.run.smithi106.stdout:CentOS-9 - BaseOS 2.8 MB/s | 2.0 MB 00:00 2024-03-10T00:37:19.279 INFO:teuthology.orchestra.run.smithi106.stdout:CentOS Stream 9 - BaseOS 8.2 MB/s | 8.0 MB 00:00 2024-03-10T00:37:20.942 INFO:teuthology.orchestra.run.smithi170.stdout:CentOS Stream 9 - BaseOS 2.8 MB/s | 8.0 MB 00:02 2024-03-10T00:37:23.818 INFO:teuthology.orchestra.run.smithi170.stdout:CentOS Stream 9 - AppStream 10 MB/s | 19 MB 00:01 2024-03-10T00:37:28.718 INFO:teuthology.orchestra.run.smithi170.stdout:CentOS Stream 9 - CRB 4.5 MB/s | 6.1 MB 00:01 2024-03-10T00:37:30.194 INFO:teuthology.orchestra.run.smithi170.stdout:CentOS Stream 9 - Extras packages 58 kB/s | 15 kB 00:00 2024-03-10T00:37:31.607 INFO:teuthology.orchestra.run.smithi170.stdout:Extra Packages for Enterprise Linux 16 MB/s | 21 MB 00:01 2024-03-10T00:37:36.545 INFO:teuthology.orchestra.run.smithi170.stdout:lab-extras 13 kB/s | 1.7 kB 00:00 2024-03-10T00:37:38.694 INFO:teuthology.orchestra.run.smithi170.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-03-10T00:37:38.694 INFO:teuthology.orchestra.run.smithi170.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2024-03-10T00:37:38.793 INFO:teuthology.orchestra.run.smithi170.stdout:Dependencies resolved. 2024-03-10T00:37:38.800 INFO:teuthology.orchestra.run.smithi170.stdout:================================================================================ 2024-03-10T00:37:38.801 INFO:teuthology.orchestra.run.smithi170.stdout: Package Architecture Version Repository Size 2024-03-10T00:37:38.801 INFO:teuthology.orchestra.run.smithi170.stdout:================================================================================ 2024-03-10T00:37:38.801 INFO:teuthology.orchestra.run.smithi170.stdout:Installing: 2024-03-10T00:37:38.801 INFO:teuthology.orchestra.run.smithi170.stdout: kernel x86_64 5.14.0-427.el9 baseos 6.0 M 2024-03-10T00:37:38.801 INFO:teuthology.orchestra.run.smithi170.stdout:Installing dependencies: 2024-03-10T00:37:38.801 INFO:teuthology.orchestra.run.smithi170.stdout: kernel-core x86_64 5.14.0-427.el9 baseos 21 M 2024-03-10T00:37:38.801 INFO:teuthology.orchestra.run.smithi170.stdout: kernel-modules x86_64 5.14.0-427.el9 baseos 39 M 2024-03-10T00:37:38.801 INFO:teuthology.orchestra.run.smithi170.stdout: kernel-modules-core x86_64 5.14.0-427.el9 baseos 33 M 2024-03-10T00:37:38.801 INFO:teuthology.orchestra.run.smithi170.stdout: 2024-03-10T00:37:38.801 INFO:teuthology.orchestra.run.smithi170.stdout:Transaction Summary 2024-03-10T00:37:38.801 INFO:teuthology.orchestra.run.smithi170.stdout:================================================================================ 2024-03-10T00:37:38.801 INFO:teuthology.orchestra.run.smithi170.stdout:Install 4 Packages 2024-03-10T00:37:38.801 INFO:teuthology.orchestra.run.smithi170.stdout: 2024-03-10T00:37:38.802 INFO:teuthology.orchestra.run.smithi170.stdout:Total download size: 100 M 2024-03-10T00:37:38.802 INFO:teuthology.orchestra.run.smithi170.stdout:Installed size: 123 M 2024-03-10T00:37:38.802 INFO:teuthology.orchestra.run.smithi170.stdout:Downloading Packages: 2024-03-10T00:37:39.780 INFO:teuthology.orchestra.run.smithi170.stdout:(1/4): kernel-5.14.0-427.el9.x86_64.rpm 6.6 MB/s | 6.0 MB 00:00 2024-03-10T00:37:43.646 INFO:teuthology.orchestra.run.smithi170.stdout:(2/4): kernel-core-5.14.0-427.el9.x86_64.rpm 4.4 MB/s | 21 MB 00:04 2024-03-10T00:37:45.005 INFO:teuthology.orchestra.run.smithi170.stdout:(3/4): kernel-modules-5.14.0-427.el9.x86_64.rpm 6.4 MB/s | 39 MB 00:06 2024-03-10T00:37:45.963 INFO:teuthology.orchestra.run.smithi170.stdout:(4/4): kernel-modules-core-5.14.0-427.el9.x86_6 5.4 MB/s | 33 MB 00:06 2024-03-10T00:37:45.964 INFO:teuthology.orchestra.run.smithi170.stdout:-------------------------------------------------------------------------------- 2024-03-10T00:37:45.964 INFO:teuthology.orchestra.run.smithi170.stdout:Total 14 MB/s | 100 MB 00:07 2024-03-10T00:37:46.848 INFO:teuthology.orchestra.run.smithi170.stdout:Running transaction check 2024-03-10T00:37:47.213 INFO:teuthology.orchestra.run.smithi170.stdout:Transaction check succeeded. 2024-03-10T00:37:47.213 INFO:teuthology.orchestra.run.smithi170.stdout:Running transaction test 2024-03-10T00:37:48.850 INFO:teuthology.orchestra.run.smithi170.stdout:Transaction test succeeded. 2024-03-10T00:37:48.850 INFO:teuthology.orchestra.run.smithi170.stdout:Running transaction 2024-03-10T00:37:51.563 INFO:teuthology.orchestra.run.smithi170.stdout: Preparing : 1/1 2024-03-10T00:37:51.702 INFO:teuthology.orchestra.run.smithi106.stdout:CentOS Stream 9 - AppStream 624 kB/s | 19 MB 00:30 2024-03-10T00:37:52.211 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : kernel-modules-core-5.14.0-427.el9.x86_64 1/4 2024-03-10T00:37:52.348 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : kernel-core-5.14.0-427.el9.x86_64 2/4 2024-03-10T00:37:55.139 INFO:teuthology.orchestra.run.smithi170.stdout: Running scriptlet: kernel-core-5.14.0-427.el9.x86_64 2/4 2024-03-10T00:37:55.308 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : kernel-modules-5.14.0-427.el9.x86_64 3/4 2024-03-10T00:37:55.761 INFO:teuthology.orchestra.run.smithi106.stdout:CentOS Stream 9 - CRB 14 MB/s | 6.1 MB 00:00 2024-03-10T00:37:57.070 INFO:teuthology.orchestra.run.smithi106.stdout:CentOS Stream 9 - Extras packages 83 kB/s | 15 kB 00:00 2024-03-10T00:37:58.672 INFO:teuthology.orchestra.run.smithi106.stdout:Extra Packages for Enterprise Linux 14 MB/s | 21 MB 00:01 2024-03-10T00:38:00.050 INFO:teuthology.orchestra.run.smithi170.stdout: Running scriptlet: kernel-modules-5.14.0-427.el9.x86_64 3/4 2024-03-10T00:38:00.237 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : kernel-5.14.0-427.el9.x86_64 4/4 2024-03-10T00:38:03.730 INFO:teuthology.orchestra.run.smithi106.stdout:lab-extras 14 kB/s | 1.7 kB 00:00 2024-03-10T00:38:04.973 INFO:teuthology.orchestra.run.smithi170.stdout: Running scriptlet: kernel-modules-core-5.14.0-427.el9.x86_64 4/4 2024-03-10T00:38:05.903 INFO:teuthology.orchestra.run.smithi106.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-03-10T00:38:05.903 INFO:teuthology.orchestra.run.smithi106.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2024-03-10T00:38:06.016 INFO:teuthology.orchestra.run.smithi106.stdout:Dependencies resolved. 2024-03-10T00:38:06.028 INFO:teuthology.orchestra.run.smithi106.stdout:================================================================================ 2024-03-10T00:38:06.028 INFO:teuthology.orchestra.run.smithi106.stdout: Package Architecture Version Repository Size 2024-03-10T00:38:06.028 INFO:teuthology.orchestra.run.smithi106.stdout:================================================================================ 2024-03-10T00:38:06.028 INFO:teuthology.orchestra.run.smithi106.stdout:Installing: 2024-03-10T00:38:06.028 INFO:teuthology.orchestra.run.smithi106.stdout: kernel x86_64 5.14.0-427.el9 baseos 6.0 M 2024-03-10T00:38:06.028 INFO:teuthology.orchestra.run.smithi106.stdout:Installing dependencies: 2024-03-10T00:38:06.028 INFO:teuthology.orchestra.run.smithi106.stdout: kernel-core x86_64 5.14.0-427.el9 baseos 21 M 2024-03-10T00:38:06.028 INFO:teuthology.orchestra.run.smithi106.stdout: kernel-modules x86_64 5.14.0-427.el9 baseos 39 M 2024-03-10T00:38:06.028 INFO:teuthology.orchestra.run.smithi106.stdout: kernel-modules-core x86_64 5.14.0-427.el9 baseos 33 M 2024-03-10T00:38:06.028 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-03-10T00:38:06.028 INFO:teuthology.orchestra.run.smithi106.stdout:Transaction Summary 2024-03-10T00:38:06.029 INFO:teuthology.orchestra.run.smithi106.stdout:================================================================================ 2024-03-10T00:38:06.029 INFO:teuthology.orchestra.run.smithi106.stdout:Install 4 Packages 2024-03-10T00:38:06.029 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-03-10T00:38:06.029 INFO:teuthology.orchestra.run.smithi106.stdout:Total download size: 100 M 2024-03-10T00:38:06.029 INFO:teuthology.orchestra.run.smithi106.stdout:Installed size: 123 M 2024-03-10T00:38:06.029 INFO:teuthology.orchestra.run.smithi106.stdout:Downloading Packages: 2024-03-10T00:38:07.931 INFO:teuthology.orchestra.run.smithi106.stdout:(1/4): kernel-5.14.0-427.el9.x86_64.rpm 3.7 MB/s | 6.0 MB 00:01 2024-03-10T00:38:10.665 INFO:teuthology.orchestra.run.smithi106.stdout:(2/4): kernel-modules-5.14.0-427.el9.x86_64.rpm 9.0 MB/s | 39 MB 00:04 2024-03-10T00:38:11.131 INFO:teuthology.orchestra.run.smithi106.stdout:(3/4): kernel-core-5.14.0-427.el9.x86_64.rpm 4.3 MB/s | 21 MB 00:04 2024-03-10T00:38:12.673 INFO:teuthology.orchestra.run.smithi106.stdout:(4/4): kernel-modules-core-5.14.0-427.el9.x86_6 7.0 MB/s | 33 MB 00:04 2024-03-10T00:38:12.673 INFO:teuthology.orchestra.run.smithi106.stdout:-------------------------------------------------------------------------------- 2024-03-10T00:38:12.673 INFO:teuthology.orchestra.run.smithi106.stdout:Total 15 MB/s | 100 MB 00:06 2024-03-10T00:38:13.559 INFO:teuthology.orchestra.run.smithi106.stdout:Running transaction check 2024-03-10T00:38:13.951 INFO:teuthology.orchestra.run.smithi106.stdout:Transaction check succeeded. 2024-03-10T00:38:13.951 INFO:teuthology.orchestra.run.smithi106.stdout:Running transaction test 2024-03-10T00:38:15.583 INFO:teuthology.orchestra.run.smithi106.stdout:Transaction test succeeded. 2024-03-10T00:38:15.583 INFO:teuthology.orchestra.run.smithi106.stdout:Running transaction 2024-03-10T00:38:18.559 INFO:teuthology.orchestra.run.smithi106.stdout: Preparing : 1/1 2024-03-10T00:38:19.228 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : kernel-modules-core-5.14.0-427.el9.x86_64 1/4 2024-03-10T00:38:19.365 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : kernel-core-5.14.0-427.el9.x86_64 2/4 2024-03-10T00:38:22.130 INFO:teuthology.orchestra.run.smithi106.stdout: Running scriptlet: kernel-core-5.14.0-427.el9.x86_64 2/4 2024-03-10T00:38:22.319 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : kernel-modules-5.14.0-427.el9.x86_64 3/4 2024-03-10T00:38:27.061 INFO:teuthology.orchestra.run.smithi106.stdout: Running scriptlet: kernel-modules-5.14.0-427.el9.x86_64 3/4 2024-03-10T00:38:27.187 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : kernel-5.14.0-427.el9.x86_64 4/4 2024-03-10T00:38:31.939 INFO:teuthology.orchestra.run.smithi106.stdout: Running scriptlet: kernel-modules-core-5.14.0-427.el9.x86_64 4/4 2024-03-10T00:38:34.139 INFO:teuthology.orchestra.run.smithi170.stdout: Running scriptlet: kernel-core-5.14.0-427.el9.x86_64 4/4 2024-03-10T00:38:34.139 INFO:teuthology.orchestra.run.smithi170.stdout:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-03-10T00:38:34.139 INFO:teuthology.orchestra.run.smithi170.stdout: 2024-03-10T00:38:34.224 INFO:teuthology.orchestra.run.smithi170.stdout: Running scriptlet: kernel-modules-5.14.0-427.el9.x86_64 4/4 2024-03-10T00:38:34.628 INFO:teuthology.orchestra.run.smithi170.stdout: Running scriptlet: kernel-5.14.0-427.el9.x86_64 4/4 2024-03-10T00:38:34.628 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : kernel-5.14.0-427.el9.x86_64 1/4 2024-03-10T00:38:34.628 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : kernel-core-5.14.0-427.el9.x86_64 2/4 2024-03-10T00:38:34.628 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : kernel-modules-5.14.0-427.el9.x86_64 3/4 2024-03-10T00:38:35.018 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : kernel-modules-core-5.14.0-427.el9.x86_64 4/4 2024-03-10T00:38:35.018 INFO:teuthology.orchestra.run.smithi170.stdout: 2024-03-10T00:38:35.018 INFO:teuthology.orchestra.run.smithi170.stdout:Installed: 2024-03-10T00:38:35.018 INFO:teuthology.orchestra.run.smithi170.stdout: kernel-5.14.0-427.el9.x86_64 2024-03-10T00:38:35.018 INFO:teuthology.orchestra.run.smithi170.stdout: kernel-core-5.14.0-427.el9.x86_64 2024-03-10T00:38:35.018 INFO:teuthology.orchestra.run.smithi170.stdout: kernel-modules-5.14.0-427.el9.x86_64 2024-03-10T00:38:35.018 INFO:teuthology.orchestra.run.smithi170.stdout: kernel-modules-core-5.14.0-427.el9.x86_64 2024-03-10T00:38:35.019 INFO:teuthology.orchestra.run.smithi170.stdout: 2024-03-10T00:38:35.019 INFO:teuthology.orchestra.run.smithi170.stdout:Complete! 2024-03-10T00:38:35.504 DEBUG:teuthology.orchestra.run.smithi170:> rpm -q kernel | sort -rV | head -n 1 2024-03-10T00:38:35.642 INFO:teuthology.orchestra.run.smithi170.stdout:kernel-5.14.0-427.el9.x86_64 2024-03-10T00:38:35.642 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-427.el9.x86_64 2024-03-10T00:38:35.642 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-427.el9.x86_64 2024-03-10T00:38:35.642 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-03-10T00:38:35.642 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-03-10T00:38:35.642 INFO:teuthology.task.kernel:Installing distro kernel on mon.b... 2024-03-10T00:38:35.643 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi170.front.sepia.ceph.com, path=None, version=distro) 2024-03-10T00:38:35.643 DEBUG:teuthology.orchestra.run.smithi170:> rpm -q kernel | sort -rV | head -n 1 2024-03-10T00:38:35.762 INFO:teuthology.orchestra.run.smithi170.stdout:kernel-5.14.0-427.el9.x86_64 2024-03-10T00:38:35.763 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-427.el9.x86_64 2024-03-10T00:38:35.763 DEBUG:teuthology.orchestra.run.smithi170:> sudo rpm -qi grub2-tools 2024-03-10T00:38:35.807 INFO:teuthology.orchestra.run.smithi170.stdout:Name : grub2-tools 2024-03-10T00:38:35.807 INFO:teuthology.orchestra.run.smithi170.stdout:Epoch : 1 2024-03-10T00:38:35.807 INFO:teuthology.orchestra.run.smithi170.stdout:Version : 2.06 2024-03-10T00:38:35.807 INFO:teuthology.orchestra.run.smithi170.stdout:Release : 68.el9 2024-03-10T00:38:35.807 INFO:teuthology.orchestra.run.smithi170.stdout:Architecture: x86_64 2024-03-10T00:38:35.807 INFO:teuthology.orchestra.run.smithi170.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-03-10T00:38:35.807 INFO:teuthology.orchestra.run.smithi170.stdout:Group : Unspecified 2024-03-10T00:38:35.807 INFO:teuthology.orchestra.run.smithi170.stdout:Size : 8264254 2024-03-10T00:38:35.807 INFO:teuthology.orchestra.run.smithi170.stdout:License : GPLv3+ 2024-03-10T00:38:35.807 INFO:teuthology.orchestra.run.smithi170.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-03-10T00:38:35.807 INFO:teuthology.orchestra.run.smithi170.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-03-10T00:38:35.807 INFO:teuthology.orchestra.run.smithi170.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-03-10T00:38:35.808 INFO:teuthology.orchestra.run.smithi170.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-03-10T00:38:35.808 INFO:teuthology.orchestra.run.smithi170.stdout:Packager : builder@centos.org 2024-03-10T00:38:35.808 INFO:teuthology.orchestra.run.smithi170.stdout:Vendor : CentOS 2024-03-10T00:38:35.808 INFO:teuthology.orchestra.run.smithi170.stdout:URL : http://www.gnu.org/software/grub/ 2024-03-10T00:38:35.808 INFO:teuthology.orchestra.run.smithi170.stdout:Summary : Support tools for GRUB. 2024-03-10T00:38:35.808 INFO:teuthology.orchestra.run.smithi170.stdout:Description : 2024-03-10T00:38:35.808 INFO:teuthology.orchestra.run.smithi170.stdout: 2024-03-10T00:38:35.808 INFO:teuthology.orchestra.run.smithi170.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-03-10T00:38:35.808 INFO:teuthology.orchestra.run.smithi170.stdout:customizable bootloader with modular architecture. It supports a rich 2024-03-10T00:38:35.808 INFO:teuthology.orchestra.run.smithi170.stdout:variety of kernel formats, file systems, computer architectures and 2024-03-10T00:38:35.808 INFO:teuthology.orchestra.run.smithi170.stdout:hardware devices. 2024-03-10T00:38:35.808 INFO:teuthology.orchestra.run.smithi170.stdout: 2024-03-10T00:38:35.808 INFO:teuthology.orchestra.run.smithi170.stdout:This subpackage provides tools for support of all platforms. 2024-03-10T00:38:35.809 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-03-10T00:38:35.809 INFO:teuthology.task.kernel:Updating grub on smithi170 to boot 5.14.0-427.el9.x86_64 2024-03-10T00:38:35.809 DEBUG:teuthology.orchestra.run.smithi170:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-03-10T00:38:36.353 INFO:teuthology.orchestra.run.smithi170.stderr:Generating grub configuration file ... 2024-03-10T00:38:37.819 INFO:teuthology.orchestra.run.smithi170.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-03-10T00:38:37.856 INFO:teuthology.orchestra.run.smithi170.stderr:done 2024-03-10T00:38:37.858 DEBUG:teuthology.orchestra.run.smithi170:> mktemp 2024-03-10T00:38:37.873 INFO:teuthology.orchestra.run.smithi170.stdout:/tmp/tmp.qNJqDgLo53 2024-03-10T00:38:37.873 DEBUG:teuthology.orchestra.run.smithi170:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.qNJqDgLo53 2024-03-10T00:38:37.942 DEBUG:teuthology.orchestra.run.smithi170:> sudo chmod 0666 /tmp/tmp.qNJqDgLo53 2024-03-10T00:38:38.080 DEBUG:teuthology.orchestra.remote:smithi170:/tmp/tmp.qNJqDgLo53 is 6KB 2024-03-10T00:38:38.100 DEBUG:teuthology.orchestra.run.smithi170:> rm -fr /tmp/tmp.qNJqDgLo53 2024-03-10T00:38:38.114 DEBUG:teuthology.orchestra.run.smithi170:> sudo /bin/ls /boot/loader/entries || true 2024-03-10T00:38:38.184 INFO:teuthology.orchestra.run.smithi170.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-03-10T00:38:38.184 INFO:teuthology.orchestra.run.smithi170.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-03-10T00:38:38.185 INFO:teuthology.orchestra.run.smithi170.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-03-10T00:38:38.185 INFO:teuthology.orchestra.run.smithi170.stdout:15949e3c88704300882482eddfd7baae-5.14.0-427.el9.x86_64.conf 2024-03-10T00:38:38.186 DEBUG:teuthology.orchestra.run.smithi170:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-427.el9.x86_64 2024-03-10T00:38:38.349 DEBUG:teuthology.orchestra.run.smithi170:> sudo shutdown -r now 2024-03-10T00:39:01.181 INFO:teuthology.orchestra.run.smithi106.stdout: Running scriptlet: kernel-core-5.14.0-427.el9.x86_64 4/4 2024-03-10T00:39:01.181 INFO:teuthology.orchestra.run.smithi106.stdout:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-03-10T00:39:01.181 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-03-10T00:39:01.264 INFO:teuthology.orchestra.run.smithi106.stdout: Running scriptlet: kernel-modules-5.14.0-427.el9.x86_64 4/4 2024-03-10T00:39:01.707 INFO:teuthology.orchestra.run.smithi106.stdout: Running scriptlet: kernel-5.14.0-427.el9.x86_64 4/4 2024-03-10T00:39:01.707 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : kernel-5.14.0-427.el9.x86_64 1/4 2024-03-10T00:39:01.707 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : kernel-core-5.14.0-427.el9.x86_64 2/4 2024-03-10T00:39:01.707 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : kernel-modules-5.14.0-427.el9.x86_64 3/4 2024-03-10T00:39:02.130 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : kernel-modules-core-5.14.0-427.el9.x86_64 4/4 2024-03-10T00:39:02.130 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-03-10T00:39:02.130 INFO:teuthology.orchestra.run.smithi106.stdout:Installed: 2024-03-10T00:39:02.130 INFO:teuthology.orchestra.run.smithi106.stdout: kernel-5.14.0-427.el9.x86_64 2024-03-10T00:39:02.131 INFO:teuthology.orchestra.run.smithi106.stdout: kernel-core-5.14.0-427.el9.x86_64 2024-03-10T00:39:02.131 INFO:teuthology.orchestra.run.smithi106.stdout: kernel-modules-5.14.0-427.el9.x86_64 2024-03-10T00:39:02.131 INFO:teuthology.orchestra.run.smithi106.stdout: kernel-modules-core-5.14.0-427.el9.x86_64 2024-03-10T00:39:02.131 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-03-10T00:39:02.131 INFO:teuthology.orchestra.run.smithi106.stdout:Complete! 2024-03-10T00:39:02.641 DEBUG:teuthology.orchestra.run.smithi106:> rpm -q kernel | sort -rV | head -n 1 2024-03-10T00:39:02.783 INFO:teuthology.orchestra.run.smithi106.stdout:kernel-5.14.0-427.el9.x86_64 2024-03-10T00:39:02.784 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-427.el9.x86_64 2024-03-10T00:39:02.784 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-427.el9.x86_64 2024-03-10T00:39:02.784 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-03-10T00:39:02.784 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-03-10T00:39:02.784 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2024-03-10T00:39:02.784 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi106.front.sepia.ceph.com, path=None, version=distro) 2024-03-10T00:39:02.784 DEBUG:teuthology.orchestra.run.smithi106:> rpm -q kernel | sort -rV | head -n 1 2024-03-10T00:39:02.904 INFO:teuthology.orchestra.run.smithi106.stdout:kernel-5.14.0-427.el9.x86_64 2024-03-10T00:39:02.905 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-427.el9.x86_64 2024-03-10T00:39:02.905 DEBUG:teuthology.orchestra.run.smithi106:> sudo rpm -qi grub2-tools 2024-03-10T00:39:02.948 INFO:teuthology.orchestra.run.smithi106.stdout:Name : grub2-tools 2024-03-10T00:39:02.948 INFO:teuthology.orchestra.run.smithi106.stdout:Epoch : 1 2024-03-10T00:39:02.948 INFO:teuthology.orchestra.run.smithi106.stdout:Version : 2.06 2024-03-10T00:39:02.948 INFO:teuthology.orchestra.run.smithi106.stdout:Release : 68.el9 2024-03-10T00:39:02.948 INFO:teuthology.orchestra.run.smithi106.stdout:Architecture: x86_64 2024-03-10T00:39:02.949 INFO:teuthology.orchestra.run.smithi106.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-03-10T00:39:02.949 INFO:teuthology.orchestra.run.smithi106.stdout:Group : Unspecified 2024-03-10T00:39:02.949 INFO:teuthology.orchestra.run.smithi106.stdout:Size : 8264254 2024-03-10T00:39:02.949 INFO:teuthology.orchestra.run.smithi106.stdout:License : GPLv3+ 2024-03-10T00:39:02.949 INFO:teuthology.orchestra.run.smithi106.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-03-10T00:39:02.949 INFO:teuthology.orchestra.run.smithi106.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-03-10T00:39:02.949 INFO:teuthology.orchestra.run.smithi106.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-03-10T00:39:02.949 INFO:teuthology.orchestra.run.smithi106.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-03-10T00:39:02.949 INFO:teuthology.orchestra.run.smithi106.stdout:Packager : builder@centos.org 2024-03-10T00:39:02.949 INFO:teuthology.orchestra.run.smithi106.stdout:Vendor : CentOS 2024-03-10T00:39:02.949 INFO:teuthology.orchestra.run.smithi106.stdout:URL : http://www.gnu.org/software/grub/ 2024-03-10T00:39:02.949 INFO:teuthology.orchestra.run.smithi106.stdout:Summary : Support tools for GRUB. 2024-03-10T00:39:02.949 INFO:teuthology.orchestra.run.smithi106.stdout:Description : 2024-03-10T00:39:02.949 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-03-10T00:39:02.949 INFO:teuthology.orchestra.run.smithi106.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-03-10T00:39:02.950 INFO:teuthology.orchestra.run.smithi106.stdout:customizable bootloader with modular architecture. It supports a rich 2024-03-10T00:39:02.950 INFO:teuthology.orchestra.run.smithi106.stdout:variety of kernel formats, file systems, computer architectures and 2024-03-10T00:39:02.950 INFO:teuthology.orchestra.run.smithi106.stdout:hardware devices. 2024-03-10T00:39:02.950 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-03-10T00:39:02.950 INFO:teuthology.orchestra.run.smithi106.stdout:This subpackage provides tools for support of all platforms. 2024-03-10T00:39:02.951 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-03-10T00:39:02.951 INFO:teuthology.task.kernel:Updating grub on smithi106 to boot 5.14.0-427.el9.x86_64 2024-03-10T00:39:02.951 DEBUG:teuthology.orchestra.run.smithi106:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-03-10T00:39:03.519 INFO:teuthology.orchestra.run.smithi106.stderr:Generating grub configuration file ... 2024-03-10T00:39:04.822 INFO:teuthology.orchestra.run.smithi106.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-03-10T00:39:04.860 INFO:teuthology.orchestra.run.smithi106.stderr:done 2024-03-10T00:39:04.866 DEBUG:teuthology.orchestra.run.smithi106:> mktemp 2024-03-10T00:39:04.881 INFO:teuthology.orchestra.run.smithi106.stdout:/tmp/tmp.zcMbijxJRm 2024-03-10T00:39:04.881 DEBUG:teuthology.orchestra.run.smithi106:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.zcMbijxJRm 2024-03-10T00:39:04.949 DEBUG:teuthology.orchestra.run.smithi106:> sudo chmod 0666 /tmp/tmp.zcMbijxJRm 2024-03-10T00:39:05.085 DEBUG:teuthology.orchestra.remote:smithi106:/tmp/tmp.zcMbijxJRm is 6KB 2024-03-10T00:39:05.134 DEBUG:teuthology.orchestra.run.smithi106:> rm -fr /tmp/tmp.zcMbijxJRm 2024-03-10T00:39:05.148 DEBUG:teuthology.orchestra.run.smithi106:> sudo /bin/ls /boot/loader/entries || true 2024-03-10T00:39:05.220 INFO:teuthology.orchestra.run.smithi106.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-03-10T00:39:05.220 INFO:teuthology.orchestra.run.smithi106.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-03-10T00:39:05.220 INFO:teuthology.orchestra.run.smithi106.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-03-10T00:39:05.220 INFO:teuthology.orchestra.run.smithi106.stdout:15949e3c88704300882482eddfd7baae-5.14.0-427.el9.x86_64.conf 2024-03-10T00:39:05.221 DEBUG:teuthology.orchestra.run.smithi106:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-427.el9.x86_64 2024-03-10T00:39:05.376 DEBUG:teuthology.orchestra.run.smithi106:> sudo shutdown -r now 2024-03-10T00:39:08.354 INFO:teuthology.task.kernel:Checking client mon.b for new kernel version... 2024-03-10T00:39:08.355 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi170.front.sepia.ceph.com' 2024-03-10T00:39:08.356 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi170.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-10T00:39:26.751 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.170 2024-03-10T00:39:35.386 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2024-03-10T00:39:35.386 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi106.front.sepia.ceph.com' 2024-03-10T00:39:35.387 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi106.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-10T00:39:35.752 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi170.front.sepia.ceph.com' 2024-03-10T00:39:35.753 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi170.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-10T00:39:38.814 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.170 2024-03-10T00:39:50.827 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi170.front.sepia.ceph.com' 2024-03-10T00:39:50.828 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi170.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-10T00:39:53.890 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.106 2024-03-10T00:40:02.898 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi106.front.sepia.ceph.com' 2024-03-10T00:40:02.899 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi106.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-10T00:40:05.950 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.106 2024-03-10T00:40:17.962 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi106.front.sepia.ceph.com' 2024-03-10T00:40:17.964 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi106.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-10T00:40:22.671 DEBUG:teuthology.orchestra.run.smithi170:> true 2024-03-10T00:40:23.076 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi170.front.sepia.ceph.com' 2024-03-10T00:40:23.076 INFO:teuthology.task.kernel:Checking kernel version of mon.b, want "5.14.0-427.el9.x86_64"... 2024-03-10T00:40:23.076 DEBUG:teuthology.orchestra.run.smithi170:> uname -r 2024-03-10T00:40:23.090 INFO:teuthology.orchestra.run.smithi170.stdout:5.14.0-427.el9.x86_64 2024-03-10T00:40:23.090 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-427.el9.x86_64 vs 5.14.0-427.el9.x86_64 2024-03-10T00:40:23.090 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-03-10T00:40:23.090 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-03-10T00:40:24.091 INFO:teuthology.task.kernel:Enabling kdb on mon.b... 2024-03-10T00:40:24.091 DEBUG:teuthology.orchestra.run.smithi170:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-03-10T00:40:24.170 INFO:teuthology.orchestra.run.smithi170.stdout:ttyS1 2024-03-10T00:40:24.226 DEBUG:teuthology.parallel:result is None 2024-03-10T00:40:36.382 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.106 2024-03-10T00:40:51.384 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi106.front.sepia.ceph.com' 2024-03-10T00:40:51.385 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi106.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-10T00:40:51.783 DEBUG:teuthology.orchestra.run.smithi106:> true 2024-03-10T00:40:52.179 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi106.front.sepia.ceph.com' 2024-03-10T00:40:52.179 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "5.14.0-427.el9.x86_64"... 2024-03-10T00:40:52.179 DEBUG:teuthology.orchestra.run.smithi106:> uname -r 2024-03-10T00:40:52.193 INFO:teuthology.orchestra.run.smithi106.stdout:5.14.0-427.el9.x86_64 2024-03-10T00:40:52.194 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-427.el9.x86_64 vs 5.14.0-427.el9.x86_64 2024-03-10T00:40:52.194 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-03-10T00:40:52.194 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-03-10T00:40:53.195 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2024-03-10T00:40:53.195 DEBUG:teuthology.orchestra.run.smithi106:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-03-10T00:40:53.259 INFO:teuthology.orchestra.run.smithi106.stdout:ttyS1 2024-03-10T00:40:53.309 DEBUG:teuthology.parallel:result is None 2024-03-10T00:40:53.309 INFO:teuthology.run_tasks:Running task internal.base... 2024-03-10T00:40:53.318 INFO:teuthology.task.internal:Creating test directory... 2024-03-10T00:40:53.318 DEBUG:teuthology.orchestra.run.smithi106:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-03-10T00:40:53.321 DEBUG:teuthology.orchestra.run.smithi170:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-03-10T00:40:53.340 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-03-10T00:40:53.347 INFO:teuthology.run_tasks:Running task internal.archive... 2024-03-10T00:40:53.353 INFO:teuthology.task.internal:Creating archive directory... 2024-03-10T00:40:53.354 DEBUG:teuthology.orchestra.run.smithi106:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-03-10T00:40:53.375 DEBUG:teuthology.orchestra.run.smithi170:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-03-10T00:40:53.446 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-03-10T00:40:53.453 INFO:teuthology.task.internal:Enabling coredump saving... 2024-03-10T00:40:53.453 DEBUG:teuthology.orchestra.run.smithi106:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-03-10T00:40:53.455 DEBUG:teuthology.orchestra.run.smithi170:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-03-10T00:40:53.484 INFO:teuthology.orchestra.run.smithi106.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-10T00:40:53.495 INFO:teuthology.orchestra.run.smithi170.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-10T00:40:53.497 INFO:teuthology.orchestra.run.smithi106.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-10T00:40:53.509 INFO:teuthology.orchestra.run.smithi170.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-10T00:40:53.511 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-03-10T00:40:53.518 INFO:teuthology.task.internal:Configuring sudo... 2024-03-10T00:40:53.518 DEBUG:teuthology.orchestra.run.smithi106:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-03-10T00:40:53.541 DEBUG:teuthology.orchestra.run.smithi170:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-03-10T00:40:53.581 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-03-10T00:40:53.590 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-03-10T00:40:53.591 DEBUG:teuthology.orchestra.run.smithi106:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-03-10T00:40:53.609 DEBUG:teuthology.orchestra.run.smithi170:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-03-10T00:40:53.637 DEBUG:teuthology.orchestra.run.smithi106:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-10T00:40:53.703 DEBUG:teuthology.orchestra.run.smithi106:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-10T00:40:53.753 DEBUG:teuthology.orchestra.run.smithi106:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-10T00:40:53.830 DEBUG:teuthology.orchestra.run.smithi106:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-10T00:40:53.875 DEBUG:teuthology.orchestra.run.smithi106:> set -ex 2024-03-10T00:40:53.875 DEBUG:teuthology.orchestra.run.smithi106:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-03-10T00:40:53.944 DEBUG:teuthology.orchestra.run.smithi170:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-10T00:40:53.988 DEBUG:teuthology.orchestra.run.smithi170:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-10T00:40:54.044 DEBUG:teuthology.orchestra.run.smithi170:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-10T00:40:54.126 DEBUG:teuthology.orchestra.run.smithi170:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-10T00:40:54.176 DEBUG:teuthology.orchestra.run.smithi170:> set -ex 2024-03-10T00:40:54.176 DEBUG:teuthology.orchestra.run.smithi170:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-03-10T00:40:54.246 DEBUG:teuthology.orchestra.run.smithi106:> sudo service rsyslog restart 2024-03-10T00:40:54.248 DEBUG:teuthology.orchestra.run.smithi170:> sudo service rsyslog restart 2024-03-10T00:40:54.298 INFO:teuthology.orchestra.run.smithi106.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-10T00:40:54.387 INFO:teuthology.orchestra.run.smithi170.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-10T00:40:54.810 INFO:teuthology.run_tasks:Running task internal.timer... 2024-03-10T00:40:54.817 INFO:teuthology.task.internal:Starting timer... 2024-03-10T00:40:54.817 INFO:teuthology.run_tasks:Running task pcp... 2024-03-10T00:40:54.862 INFO:teuthology.run_tasks:Running task selinux... 2024-03-10T00:40:54.871 DEBUG:teuthology.task:Applying overrides for task selinux: {'whitelist': ['scontext=system_u:system_r:logrotate_t:s0']} 2024-03-10T00:40:54.871 DEBUG:teuthology.orchestra.run.smithi106:> sudo service auditd rotate 2024-03-10T00:40:54.934 INFO:teuthology.orchestra.run.smithi106.stdout:Rotating logs: 2024-03-10T00:40:54.935 DEBUG:teuthology.orchestra.run.smithi170:> sudo service auditd rotate 2024-03-10T00:40:54.984 INFO:teuthology.orchestra.run.smithi170.stdout:Rotating logs: 2024-03-10T00:40:54.985 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-03-10T00:40:54.986 DEBUG:teuthology.orchestra.run.smithi106:> /usr/sbin/getenforce 2024-03-10T00:40:55.008 INFO:teuthology.orchestra.run.smithi106.stdout:Permissive 2024-03-10T00:40:55.008 DEBUG:teuthology.orchestra.run.smithi170:> /usr/sbin/getenforce 2024-03-10T00:40:55.034 INFO:teuthology.orchestra.run.smithi170.stdout:Permissive 2024-03-10T00:40:55.034 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi106.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi170.front.sepia.ceph.com': 'permissive'} 2024-03-10T00:40:55.034 DEBUG:teuthology.orchestra.run.smithi106:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-10T00:40:55.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-10T00:40:55.079 DEBUG:teuthology.orchestra.run.smithi170:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-10T00:40:55.108 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-10T00:40:55.109 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-03-10T00:40:55.109 DEBUG:teuthology.orchestra.run.smithi106:> sudo /usr/sbin/setenforce permissive 2024-03-10T00:40:55.169 DEBUG:teuthology.orchestra.run.smithi170:> sudo /usr/sbin/setenforce permissive 2024-03-10T00:40:55.211 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-03-10T00:40:55.223 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-03-10T00:40:55.238 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-03-10T00:40:55.239 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi106.front.sepia.ceph.com,smithi170.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-03-10T00:45:08.371 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi106.front.sepia.ceph.com'), Remote(name='ubuntu@smithi170.front.sepia.ceph.com')] 2024-03-10T00:45:08.373 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi106.front.sepia.ceph.com' 2024-03-10T00:45:08.374 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi106.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-10T00:45:08.467 DEBUG:teuthology.orchestra.run.smithi106:> true 2024-03-10T00:45:08.544 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi106.front.sepia.ceph.com' 2024-03-10T00:45:08.544 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi170.front.sepia.ceph.com' 2024-03-10T00:45:08.545 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi170.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-10T00:45:08.636 DEBUG:teuthology.orchestra.run.smithi170:> true 2024-03-10T00:45:08.726 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi170.front.sepia.ceph.com' 2024-03-10T00:45:08.726 INFO:teuthology.run_tasks:Running task clock... 2024-03-10T00:45:08.741 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-03-10T00:45:08.741 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-03-10T00:45:08.742 DEBUG:teuthology.orchestra.run.smithi106:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-10T00:45:08.744 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-03-10T00:45:08.744 DEBUG:teuthology.orchestra.run.smithi170:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-10T00:45:08.772 INFO:teuthology.orchestra.run.smithi106.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-03-10T00:45:08.789 INFO:teuthology.orchestra.run.smithi106.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-03-10T00:45:08.798 INFO:teuthology.orchestra.run.smithi170.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-03-10T00:45:08.816 INFO:teuthology.orchestra.run.smithi170.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-03-10T00:45:08.817 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: ntpd: command not found 2024-03-10T00:45:08.831 INFO:teuthology.orchestra.run.smithi106.stdout:506 Cannot talk to daemon 2024-03-10T00:45:08.846 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: ntpd: command not found 2024-03-10T00:45:08.847 INFO:teuthology.orchestra.run.smithi106.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-03-10T00:45:08.860 INFO:teuthology.orchestra.run.smithi170.stdout:506 Cannot talk to daemon 2024-03-10T00:45:08.864 INFO:teuthology.orchestra.run.smithi106.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-03-10T00:45:08.876 INFO:teuthology.orchestra.run.smithi170.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-03-10T00:45:08.893 INFO:teuthology.orchestra.run.smithi170.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-03-10T00:45:08.911 INFO:teuthology.orchestra.run.smithi106.stderr:bash: line 1: ntpq: command not found 2024-03-10T00:45:08.914 INFO:teuthology.orchestra.run.smithi106.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-10T00:45:08.914 INFO:teuthology.orchestra.run.smithi106.stdout:=============================================================================== 2024-03-10T00:45:08.915 INFO:teuthology.orchestra.run.smithi106.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-10T00:45:08.915 INFO:teuthology.orchestra.run.smithi106.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-10T00:45:08.915 INFO:teuthology.orchestra.run.smithi106.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-10T00:45:08.915 INFO:teuthology.orchestra.run.smithi106.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-10T00:45:08.945 INFO:teuthology.orchestra.run.smithi170.stderr:bash: line 1: ntpq: command not found 2024-03-10T00:45:08.949 INFO:teuthology.orchestra.run.smithi170.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-10T00:45:08.949 INFO:teuthology.orchestra.run.smithi170.stdout:=============================================================================== 2024-03-10T00:45:08.949 INFO:teuthology.orchestra.run.smithi170.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-10T00:45:08.949 INFO:teuthology.orchestra.run.smithi170.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-10T00:45:08.949 INFO:teuthology.orchestra.run.smithi170.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-10T00:45:08.949 INFO:teuthology.orchestra.run.smithi170.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-10T00:45:08.950 INFO:teuthology.run_tasks:Running task pexec... 2024-03-10T00:45:08.960 INFO:teuthology.task.pexec:Executing custom commands... 2024-03-10T00:45:08.961 INFO:teuthology.task.pexec:Running commands on host ubuntu@smithi106.front.sepia.ceph.com 2024-03-10T00:45:08.961 DEBUG:teuthology.orchestra.run.smithi106:> TESTDIR=/home/ubuntu/cephtest bash -s 2024-03-10T00:45:08.961 INFO:teuthology.task.pexec:Running commands on host ubuntu@smithi170.front.sepia.ceph.com 2024-03-10T00:45:08.962 DEBUG:teuthology.orchestra.run.smithi170:> TESTDIR=/home/ubuntu/cephtest bash -s 2024-03-10T00:45:09.539 INFO:teuthology.orchestra.run.smithi106.stdout:Last metadata expiration check: 0:00:48 ago on Sun 10 Mar 2024 12:44:21 AM UTC. 2024-03-10T00:45:09.566 INFO:teuthology.orchestra.run.smithi170.stdout:Last metadata expiration check: 0:00:49 ago on Sun 10 Mar 2024 12:44:20 AM UTC. 2024-03-10T00:45:09.684 INFO:teuthology.orchestra.run.smithi106.stdout:Dependencies resolved. 2024-03-10T00:45:09.703 INFO:teuthology.orchestra.run.smithi106.stdout:================================================================================ 2024-03-10T00:45:09.703 INFO:teuthology.orchestra.run.smithi106.stdout: Package Architecture Version Repository Size 2024-03-10T00:45:09.703 INFO:teuthology.orchestra.run.smithi106.stdout:================================================================================ 2024-03-10T00:45:09.703 INFO:teuthology.orchestra.run.smithi106.stdout:Installing: 2024-03-10T00:45:09.703 INFO:teuthology.orchestra.run.smithi106.stdout: runc x86_64 4:1.1.12-2.el9 appstream 3.1 M 2024-03-10T00:45:09.703 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-03-10T00:45:09.703 INFO:teuthology.orchestra.run.smithi106.stdout:Transaction Summary 2024-03-10T00:45:09.704 INFO:teuthology.orchestra.run.smithi106.stdout:================================================================================ 2024-03-10T00:45:09.704 INFO:teuthology.orchestra.run.smithi106.stdout:Install 1 Package 2024-03-10T00:45:09.704 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-03-10T00:45:09.709 INFO:teuthology.orchestra.run.smithi106.stdout:Total download size: 3.1 M 2024-03-10T00:45:09.709 INFO:teuthology.orchestra.run.smithi106.stdout:Installed size: 9.8 M 2024-03-10T00:45:09.709 INFO:teuthology.orchestra.run.smithi106.stdout:Downloading Packages: 2024-03-10T00:45:09.712 INFO:teuthology.orchestra.run.smithi170.stdout:Dependencies resolved. 2024-03-10T00:45:09.714 INFO:teuthology.orchestra.run.smithi170.stdout:================================================================================ 2024-03-10T00:45:09.714 INFO:teuthology.orchestra.run.smithi170.stdout: Package Architecture Version Repository Size 2024-03-10T00:45:09.714 INFO:teuthology.orchestra.run.smithi170.stdout:================================================================================ 2024-03-10T00:45:09.714 INFO:teuthology.orchestra.run.smithi170.stdout:Installing: 2024-03-10T00:45:09.714 INFO:teuthology.orchestra.run.smithi170.stdout: runc x86_64 4:1.1.12-2.el9 appstream 3.1 M 2024-03-10T00:45:09.714 INFO:teuthology.orchestra.run.smithi170.stdout: 2024-03-10T00:45:09.714 INFO:teuthology.orchestra.run.smithi170.stdout:Transaction Summary 2024-03-10T00:45:09.714 INFO:teuthology.orchestra.run.smithi170.stdout:================================================================================ 2024-03-10T00:45:09.714 INFO:teuthology.orchestra.run.smithi170.stdout:Install 1 Package 2024-03-10T00:45:09.714 INFO:teuthology.orchestra.run.smithi170.stdout: 2024-03-10T00:45:09.727 INFO:teuthology.orchestra.run.smithi170.stdout:Total download size: 3.1 M 2024-03-10T00:45:09.727 INFO:teuthology.orchestra.run.smithi170.stdout:Installed size: 9.8 M 2024-03-10T00:45:09.727 INFO:teuthology.orchestra.run.smithi170.stdout:Downloading Packages: 2024-03-10T00:45:10.093 INFO:teuthology.orchestra.run.smithi106.stdout:runc-1.1.12-2.el9.x86_64.rpm 12 MB/s | 3.1 MB 00:00 2024-03-10T00:45:10.094 INFO:teuthology.orchestra.run.smithi106.stdout:-------------------------------------------------------------------------------- 2024-03-10T00:45:10.094 INFO:teuthology.orchestra.run.smithi106.stdout:Total 8.1 MB/s | 3.1 MB 00:00 2024-03-10T00:45:10.130 INFO:teuthology.orchestra.run.smithi106.stdout:Running transaction check 2024-03-10T00:45:10.135 INFO:teuthology.orchestra.run.smithi106.stdout:Transaction check succeeded. 2024-03-10T00:45:10.136 INFO:teuthology.orchestra.run.smithi106.stdout:Running transaction test 2024-03-10T00:45:10.214 INFO:teuthology.orchestra.run.smithi106.stdout:Transaction test succeeded. 2024-03-10T00:45:10.215 INFO:teuthology.orchestra.run.smithi106.stdout:Running transaction 2024-03-10T00:45:10.323 INFO:teuthology.orchestra.run.smithi170.stdout:runc-1.1.12-2.el9.x86_64.rpm 9.6 MB/s | 3.1 MB 00:00 2024-03-10T00:45:10.323 INFO:teuthology.orchestra.run.smithi170.stdout:-------------------------------------------------------------------------------- 2024-03-10T00:45:10.323 INFO:teuthology.orchestra.run.smithi170.stdout:Total 5.3 MB/s | 3.1 MB 00:00 2024-03-10T00:45:10.359 INFO:teuthology.orchestra.run.smithi170.stdout:Running transaction check 2024-03-10T00:45:10.364 INFO:teuthology.orchestra.run.smithi170.stdout:Transaction check succeeded. 2024-03-10T00:45:10.364 INFO:teuthology.orchestra.run.smithi170.stdout:Running transaction test 2024-03-10T00:45:10.428 INFO:teuthology.orchestra.run.smithi106.stdout: Preparing : 1/1 2024-03-10T00:45:10.442 INFO:teuthology.orchestra.run.smithi170.stdout:Transaction test succeeded. 2024-03-10T00:45:10.443 INFO:teuthology.orchestra.run.smithi170.stdout:Running transaction 2024-03-10T00:45:10.521 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : runc-4:1.1.12-2.el9.x86_64 1/1 2024-03-10T00:45:10.656 INFO:teuthology.orchestra.run.smithi170.stdout: Preparing : 1/1 2024-03-10T00:45:10.759 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : runc-4:1.1.12-2.el9.x86_64 1/1 2024-03-10T00:45:11.074 INFO:teuthology.orchestra.run.smithi106.stdout: Running scriptlet: runc-4:1.1.12-2.el9.x86_64 1/1 2024-03-10T00:45:11.371 INFO:teuthology.orchestra.run.smithi170.stdout: Running scriptlet: runc-4:1.1.12-2.el9.x86_64 1/1 2024-03-10T00:45:11.385 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : runc-4:1.1.12-2.el9.x86_64 1/1 2024-03-10T00:45:11.385 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-03-10T00:45:11.385 INFO:teuthology.orchestra.run.smithi106.stdout:Installed: 2024-03-10T00:45:11.385 INFO:teuthology.orchestra.run.smithi106.stdout: runc-4:1.1.12-2.el9.x86_64 2024-03-10T00:45:11.385 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-03-10T00:45:11.385 INFO:teuthology.orchestra.run.smithi106.stdout:Complete! 2024-03-10T00:45:11.632 DEBUG:teuthology.parallel:result is None 2024-03-10T00:45:11.673 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : runc-4:1.1.12-2.el9.x86_64 1/1 2024-03-10T00:45:11.673 INFO:teuthology.orchestra.run.smithi170.stdout: 2024-03-10T00:45:11.673 INFO:teuthology.orchestra.run.smithi170.stdout:Installed: 2024-03-10T00:45:11.673 INFO:teuthology.orchestra.run.smithi170.stdout: runc-4:1.1.12-2.el9.x86_64 2024-03-10T00:45:11.673 INFO:teuthology.orchestra.run.smithi170.stdout: 2024-03-10T00:45:11.674 INFO:teuthology.orchestra.run.smithi170.stdout:Complete! 2024-03-10T00:45:11.923 DEBUG:teuthology.parallel:result is None 2024-03-10T00:45:11.923 INFO:teuthology.run_tasks:Running task nvme_loop... 2024-03-10T00:45:11.951 INFO:tasks.nvme_loop:Setting up nvme_loop on scratch devices... 2024-03-10T00:45:11.952 DEBUG:teuthology.orchestra.run.smithi106:> set -ex 2024-03-10T00:45:11.952 DEBUG:teuthology.orchestra.run.smithi106:> dd if=/scratch_devs of=/dev/stdout 2024-03-10T00:45:11.967 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-03-10T00:45:11.967 DEBUG:teuthology.orchestra.run.smithi106:> stat /dev/vg_nvme/lv_1 2024-03-10T00:45:12.022 INFO:teuthology.orchestra.run.smithi106.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-03-10T00:45:12.022 INFO:teuthology.orchestra.run.smithi106.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-10T00:45:12.022 INFO:teuthology.orchestra.run.smithi106.stdout:Device: 5h/5d Inode: 665 Links: 1 2024-03-10T00:45:12.022 INFO:teuthology.orchestra.run.smithi106.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-10T00:45:12.022 INFO:teuthology.orchestra.run.smithi106.stdout:Context: system_u:object_r:device_t:s0 2024-03-10T00:45:12.022 INFO:teuthology.orchestra.run.smithi106.stdout:Access: 2024-03-10 00:45:07.374897874 +0000 2024-03-10T00:45:12.022 INFO:teuthology.orchestra.run.smithi106.stdout:Modify: 2024-03-10 00:44:45.009409756 +0000 2024-03-10T00:45:12.022 INFO:teuthology.orchestra.run.smithi106.stdout:Change: 2024-03-10 00:44:45.009409756 +0000 2024-03-10T00:45:12.022 INFO:teuthology.orchestra.run.smithi106.stdout: Birth: 2024-03-10 00:44:45.009409756 +0000 2024-03-10T00:45:12.022 DEBUG:teuthology.orchestra.run.smithi106:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-03-10T00:45:12.088 INFO:teuthology.orchestra.run.smithi106.stderr:1+0 records in 2024-03-10T00:45:12.088 INFO:teuthology.orchestra.run.smithi106.stderr:1+0 records out 2024-03-10T00:45:12.089 INFO:teuthology.orchestra.run.smithi106.stderr:512 bytes copied, 0.00035484 s, 1.4 MB/s 2024-03-10T00:45:12.090 DEBUG:teuthology.orchestra.run.smithi106:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-03-10T00:45:12.145 DEBUG:teuthology.orchestra.run.smithi106:> stat /dev/vg_nvme/lv_2 2024-03-10T00:45:12.199 INFO:teuthology.orchestra.run.smithi106.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-03-10T00:45:12.199 INFO:teuthology.orchestra.run.smithi106.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-10T00:45:12.199 INFO:teuthology.orchestra.run.smithi106.stdout:Device: 5h/5d Inode: 685 Links: 1 2024-03-10T00:45:12.199 INFO:teuthology.orchestra.run.smithi106.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-10T00:45:12.200 INFO:teuthology.orchestra.run.smithi106.stdout:Context: system_u:object_r:device_t:s0 2024-03-10T00:45:12.200 INFO:teuthology.orchestra.run.smithi106.stdout:Access: 2024-03-10 00:45:07.374897874 +0000 2024-03-10T00:45:12.200 INFO:teuthology.orchestra.run.smithi106.stdout:Modify: 2024-03-10 00:44:45.351401929 +0000 2024-03-10T00:45:12.200 INFO:teuthology.orchestra.run.smithi106.stdout:Change: 2024-03-10 00:44:45.351401929 +0000 2024-03-10T00:45:12.200 INFO:teuthology.orchestra.run.smithi106.stdout: Birth: 2024-03-10 00:44:45.351401929 +0000 2024-03-10T00:45:12.200 DEBUG:teuthology.orchestra.run.smithi106:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-03-10T00:45:12.271 INFO:teuthology.orchestra.run.smithi106.stderr:1+0 records in 2024-03-10T00:45:12.271 INFO:teuthology.orchestra.run.smithi106.stderr:1+0 records out 2024-03-10T00:45:12.271 INFO:teuthology.orchestra.run.smithi106.stderr:512 bytes copied, 0.00020979 s, 2.4 MB/s 2024-03-10T00:45:12.272 DEBUG:teuthology.orchestra.run.smithi106:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-03-10T00:45:12.328 DEBUG:teuthology.orchestra.run.smithi106:> stat /dev/vg_nvme/lv_3 2024-03-10T00:45:12.383 INFO:teuthology.orchestra.run.smithi106.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-03-10T00:45:12.383 INFO:teuthology.orchestra.run.smithi106.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-10T00:45:12.383 INFO:teuthology.orchestra.run.smithi106.stdout:Device: 5h/5d Inode: 703 Links: 1 2024-03-10T00:45:12.383 INFO:teuthology.orchestra.run.smithi106.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-10T00:45:12.383 INFO:teuthology.orchestra.run.smithi106.stdout:Context: system_u:object_r:device_t:s0 2024-03-10T00:45:12.383 INFO:teuthology.orchestra.run.smithi106.stdout:Access: 2024-03-10 00:45:07.374897874 +0000 2024-03-10T00:45:12.383 INFO:teuthology.orchestra.run.smithi106.stdout:Modify: 2024-03-10 00:44:45.708393758 +0000 2024-03-10T00:45:12.383 INFO:teuthology.orchestra.run.smithi106.stdout:Change: 2024-03-10 00:44:45.708393758 +0000 2024-03-10T00:45:12.384 INFO:teuthology.orchestra.run.smithi106.stdout: Birth: 2024-03-10 00:44:45.708393758 +0000 2024-03-10T00:45:12.384 DEBUG:teuthology.orchestra.run.smithi106:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-03-10T00:45:12.449 INFO:teuthology.orchestra.run.smithi106.stderr:1+0 records in 2024-03-10T00:45:12.449 INFO:teuthology.orchestra.run.smithi106.stderr:1+0 records out 2024-03-10T00:45:12.449 INFO:teuthology.orchestra.run.smithi106.stderr:512 bytes copied, 0.000207494 s, 2.5 MB/s 2024-03-10T00:45:12.450 DEBUG:teuthology.orchestra.run.smithi106:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-03-10T00:45:12.506 DEBUG:teuthology.orchestra.run.smithi106:> stat /dev/vg_nvme/lv_4 2024-03-10T00:45:12.561 INFO:teuthology.orchestra.run.smithi106.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-03-10T00:45:12.561 INFO:teuthology.orchestra.run.smithi106.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-10T00:45:12.561 INFO:teuthology.orchestra.run.smithi106.stdout:Device: 5h/5d Inode: 725 Links: 1 2024-03-10T00:45:12.562 INFO:teuthology.orchestra.run.smithi106.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-10T00:45:12.562 INFO:teuthology.orchestra.run.smithi106.stdout:Context: system_u:object_r:device_t:s0 2024-03-10T00:45:12.562 INFO:teuthology.orchestra.run.smithi106.stdout:Access: 2024-03-10 00:45:07.374897874 +0000 2024-03-10T00:45:12.562 INFO:teuthology.orchestra.run.smithi106.stdout:Modify: 2024-03-10 00:44:46.039386183 +0000 2024-03-10T00:45:12.562 INFO:teuthology.orchestra.run.smithi106.stdout:Change: 2024-03-10 00:44:46.039386183 +0000 2024-03-10T00:45:12.562 INFO:teuthology.orchestra.run.smithi106.stdout: Birth: 2024-03-10 00:44:46.039386183 +0000 2024-03-10T00:45:12.563 DEBUG:teuthology.orchestra.run.smithi106:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-03-10T00:45:12.627 INFO:teuthology.orchestra.run.smithi106.stderr:1+0 records in 2024-03-10T00:45:12.627 INFO:teuthology.orchestra.run.smithi106.stderr:1+0 records out 2024-03-10T00:45:12.627 INFO:teuthology.orchestra.run.smithi106.stderr:512 bytes copied, 0.000184181 s, 2.8 MB/s 2024-03-10T00:45:12.628 DEBUG:teuthology.orchestra.run.smithi106:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-03-10T00:45:12.684 DEBUG:teuthology.orchestra.run.smithi106:> grep '^nvme_loop' /proc/modules || sudo modprobe nvme_loop && sudo mkdir -p /sys/kernel/config/nvmet/hosts/hostnqn && sudo mkdir -p /sys/kernel/config/nvmet/ports/1 && echo loop | sudo tee /sys/kernel/config/nvmet/ports/1/addr_trtype 2024-03-10T00:45:12.828 INFO:teuthology.orchestra.run.smithi106.stdout:loop 2024-03-10T00:45:12.830 INFO:tasks.nvme_loop:Connecting nvme_loop smithi106:/dev/vg_nvme/lv_1... 2024-03-10T00:45:12.830 DEBUG:teuthology.orchestra.run.smithi106:> sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_1 && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_1/attr_allow_any_host && sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_1/namespaces/1 && echo -n /dev/vg_nvme/lv_1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_1/namespaces/1/device_path && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_1/namespaces/1/enable && sudo ln -s /sys/kernel/config/nvmet/subsystems/lv_1 /sys/kernel/config/nvmet/ports/1/subsystems/lv_1 && sudo nvme connect -t loop -n lv_1 -q hostnqn 2024-03-10T00:45:12.867 INFO:teuthology.orchestra.run.smithi106.stdout:1 2024-03-10T00:45:12.904 INFO:teuthology.orchestra.run.smithi106.stdout:/dev/vg_nvme/lv_11 2024-03-10T00:45:12.934 INFO:tasks.nvme_loop:Connecting nvme_loop smithi106:/dev/vg_nvme/lv_2... 2024-03-10T00:45:12.934 DEBUG:teuthology.orchestra.run.smithi106:> sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_2 && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_2/attr_allow_any_host && sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_2/namespaces/1 && echo -n /dev/vg_nvme/lv_2 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_2/namespaces/1/device_path && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_2/namespaces/1/enable && sudo ln -s /sys/kernel/config/nvmet/subsystems/lv_2 /sys/kernel/config/nvmet/ports/1/subsystems/lv_2 && sudo nvme connect -t loop -n lv_2 -q hostnqn 2024-03-10T00:45:13.025 INFO:teuthology.orchestra.run.smithi106.stdout:1 2024-03-10T00:45:13.063 INFO:teuthology.orchestra.run.smithi106.stdout:/dev/vg_nvme/lv_21 2024-03-10T00:45:13.092 INFO:tasks.nvme_loop:Connecting nvme_loop smithi106:/dev/vg_nvme/lv_3... 2024-03-10T00:45:13.093 DEBUG:teuthology.orchestra.run.smithi106:> sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_3 && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_3/attr_allow_any_host && sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_3/namespaces/1 && echo -n /dev/vg_nvme/lv_3 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_3/namespaces/1/device_path && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_3/namespaces/1/enable && sudo ln -s /sys/kernel/config/nvmet/subsystems/lv_3 /sys/kernel/config/nvmet/ports/1/subsystems/lv_3 && sudo nvme connect -t loop -n lv_3 -q hostnqn 2024-03-10T00:45:13.132 INFO:teuthology.orchestra.run.smithi106.stdout:1 2024-03-10T00:45:13.171 INFO:teuthology.orchestra.run.smithi106.stdout:/dev/vg_nvme/lv_31 2024-03-10T00:45:13.200 INFO:tasks.nvme_loop:Connecting nvme_loop smithi106:/dev/vg_nvme/lv_4... 2024-03-10T00:45:13.200 DEBUG:teuthology.orchestra.run.smithi106:> sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_4 && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_4/attr_allow_any_host && sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_4/namespaces/1 && echo -n /dev/vg_nvme/lv_4 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_4/namespaces/1/device_path && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_4/namespaces/1/enable && sudo ln -s /sys/kernel/config/nvmet/subsystems/lv_4 /sys/kernel/config/nvmet/ports/1/subsystems/lv_4 && sudo nvme connect -t loop -n lv_4 -q hostnqn 2024-03-10T00:45:13.284 INFO:teuthology.orchestra.run.smithi106.stdout:1 2024-03-10T00:45:13.323 INFO:teuthology.orchestra.run.smithi106.stdout:/dev/vg_nvme/lv_41 2024-03-10T00:45:13.360 DEBUG:teuthology.orchestra.run.smithi106:> set -ex 2024-03-10T00:45:13.360 DEBUG:teuthology.orchestra.run.smithi106:> dd if=/scratch_devs of=/dev/stdout 2024-03-10T00:45:13.414 DEBUG:teuthology.orchestra.run.smithi106:> sudo nvme list 2024-03-10T00:45:13.480 INFO:teuthology.orchestra.run.smithi106.stdout:Node SN Model Namespace Usage Format FW Rev 2024-03-10T00:45:13.480 INFO:teuthology.orchestra.run.smithi106.stdout:---------------- -------------------- ---------------------------------------- --------- -------------------------- ---------------- -------- 2024-03-10T00:45:13.480 INFO:teuthology.orchestra.run.smithi106.stdout:/dev/nvme0n1 PHFT6204013R400BGN INTEL SSDPEDMD400G4 1 400.09 GB / 400.09 GB 512 B + 0 B 8DV101H0 2024-03-10T00:45:13.480 INFO:teuthology.orchestra.run.smithi106.stdout:/dev/nvme1n1 360593095f039aa499ca Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 5.14.0-4 2024-03-10T00:45:13.480 INFO:teuthology.orchestra.run.smithi106.stdout:/dev/nvme2n1 2c097f3a9f2b69440f66 Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 5.14.0-4 2024-03-10T00:45:13.480 INFO:teuthology.orchestra.run.smithi106.stdout:/dev/nvme3n1 43400f8a9a7d42abdb0c Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 5.14.0-4 2024-03-10T00:45:13.481 INFO:teuthology.orchestra.run.smithi106.stdout:/dev/nvme4n1 907b47e64be117f399fb Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 5.14.0-4 2024-03-10T00:45:13.481 INFO:tasks.nvme_loop:new_devs ['/dev/nvme1n1', '/dev/nvme2n1', '/dev/nvme3n1', '/dev/nvme4n1'] 2024-03-10T00:45:13.482 DEBUG:teuthology.orchestra.run.smithi106:> set -ex 2024-03-10T00:45:13.482 DEBUG:teuthology.orchestra.run.smithi106:> sudo dd of=/scratch_devs 2024-03-10T00:45:13.548 DEBUG:teuthology.orchestra.run.smithi170:> set -ex 2024-03-10T00:45:13.548 DEBUG:teuthology.orchestra.run.smithi170:> dd if=/scratch_devs of=/dev/stdout 2024-03-10T00:45:13.568 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-03-10T00:45:13.568 DEBUG:teuthology.orchestra.run.smithi170:> stat /dev/vg_nvme/lv_1 2024-03-10T00:45:13.626 INFO:teuthology.orchestra.run.smithi170.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-03-10T00:45:13.626 INFO:teuthology.orchestra.run.smithi170.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-10T00:45:13.626 INFO:teuthology.orchestra.run.smithi170.stdout:Device: 5h/5d Inode: 668 Links: 1 2024-03-10T00:45:13.627 INFO:teuthology.orchestra.run.smithi170.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-10T00:45:13.627 INFO:teuthology.orchestra.run.smithi170.stdout:Context: system_u:object_r:device_t:s0 2024-03-10T00:45:13.627 INFO:teuthology.orchestra.run.smithi170.stdout:Access: 2024-03-10 00:45:07.440037454 +0000 2024-03-10T00:45:13.627 INFO:teuthology.orchestra.run.smithi170.stdout:Modify: 2024-03-10 00:44:45.002553730 +0000 2024-03-10T00:45:13.627 INFO:teuthology.orchestra.run.smithi170.stdout:Change: 2024-03-10 00:44:45.002553730 +0000 2024-03-10T00:45:13.627 INFO:teuthology.orchestra.run.smithi170.stdout: Birth: 2024-03-10 00:44:45.002553730 +0000 2024-03-10T00:45:13.627 DEBUG:teuthology.orchestra.run.smithi170:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-03-10T00:45:13.696 INFO:teuthology.orchestra.run.smithi170.stderr:1+0 records in 2024-03-10T00:45:13.697 INFO:teuthology.orchestra.run.smithi170.stderr:1+0 records out 2024-03-10T00:45:13.697 INFO:teuthology.orchestra.run.smithi170.stderr:512 bytes copied, 0.000123119 s, 4.2 MB/s 2024-03-10T00:45:13.698 DEBUG:teuthology.orchestra.run.smithi170:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-03-10T00:45:13.756 DEBUG:teuthology.orchestra.run.smithi170:> stat /dev/vg_nvme/lv_2 2024-03-10T00:45:13.814 INFO:teuthology.orchestra.run.smithi170.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-03-10T00:45:13.814 INFO:teuthology.orchestra.run.smithi170.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-10T00:45:13.814 INFO:teuthology.orchestra.run.smithi170.stdout:Device: 5h/5d Inode: 684 Links: 1 2024-03-10T00:45:13.814 INFO:teuthology.orchestra.run.smithi170.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-10T00:45:13.814 INFO:teuthology.orchestra.run.smithi170.stdout:Context: system_u:object_r:device_t:s0 2024-03-10T00:45:13.814 INFO:teuthology.orchestra.run.smithi170.stdout:Access: 2024-03-10 00:45:07.440037454 +0000 2024-03-10T00:45:13.815 INFO:teuthology.orchestra.run.smithi170.stdout:Modify: 2024-03-10 00:44:45.323546343 +0000 2024-03-10T00:45:13.815 INFO:teuthology.orchestra.run.smithi170.stdout:Change: 2024-03-10 00:44:45.323546343 +0000 2024-03-10T00:45:13.815 INFO:teuthology.orchestra.run.smithi170.stdout: Birth: 2024-03-10 00:44:45.323546343 +0000 2024-03-10T00:45:13.815 DEBUG:teuthology.orchestra.run.smithi170:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-03-10T00:45:13.882 INFO:teuthology.orchestra.run.smithi170.stderr:1+0 records in 2024-03-10T00:45:13.882 INFO:teuthology.orchestra.run.smithi170.stderr:1+0 records out 2024-03-10T00:45:13.882 INFO:teuthology.orchestra.run.smithi170.stderr:512 bytes copied, 0.000109121 s, 4.7 MB/s 2024-03-10T00:45:13.883 DEBUG:teuthology.orchestra.run.smithi170:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-03-10T00:45:13.941 DEBUG:teuthology.orchestra.run.smithi170:> stat /dev/vg_nvme/lv_3 2024-03-10T00:45:13.997 INFO:teuthology.orchestra.run.smithi170.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-03-10T00:45:13.997 INFO:teuthology.orchestra.run.smithi170.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-10T00:45:13.998 INFO:teuthology.orchestra.run.smithi170.stdout:Device: 5h/5d Inode: 705 Links: 1 2024-03-10T00:45:13.998 INFO:teuthology.orchestra.run.smithi170.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-10T00:45:13.998 INFO:teuthology.orchestra.run.smithi170.stdout:Context: system_u:object_r:device_t:s0 2024-03-10T00:45:13.998 INFO:teuthology.orchestra.run.smithi170.stdout:Access: 2024-03-10 00:45:07.440037454 +0000 2024-03-10T00:45:13.998 INFO:teuthology.orchestra.run.smithi170.stdout:Modify: 2024-03-10 00:44:45.658538635 +0000 2024-03-10T00:45:13.998 INFO:teuthology.orchestra.run.smithi170.stdout:Change: 2024-03-10 00:44:45.658538635 +0000 2024-03-10T00:45:13.998 INFO:teuthology.orchestra.run.smithi170.stdout: Birth: 2024-03-10 00:44:45.658538635 +0000 2024-03-10T00:45:13.998 DEBUG:teuthology.orchestra.run.smithi170:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-03-10T00:45:14.065 INFO:teuthology.orchestra.run.smithi170.stderr:1+0 records in 2024-03-10T00:45:14.065 INFO:teuthology.orchestra.run.smithi170.stderr:1+0 records out 2024-03-10T00:45:14.065 INFO:teuthology.orchestra.run.smithi170.stderr:512 bytes copied, 0.000106642 s, 4.8 MB/s 2024-03-10T00:45:14.066 DEBUG:teuthology.orchestra.run.smithi170:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-03-10T00:45:14.123 DEBUG:teuthology.orchestra.run.smithi170:> stat /dev/vg_nvme/lv_4 2024-03-10T00:45:14.178 INFO:teuthology.orchestra.run.smithi170.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-03-10T00:45:14.179 INFO:teuthology.orchestra.run.smithi170.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-10T00:45:14.179 INFO:teuthology.orchestra.run.smithi170.stdout:Device: 5h/5d Inode: 722 Links: 1 2024-03-10T00:45:14.179 INFO:teuthology.orchestra.run.smithi170.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-10T00:45:14.179 INFO:teuthology.orchestra.run.smithi170.stdout:Context: system_u:object_r:device_t:s0 2024-03-10T00:45:14.179 INFO:teuthology.orchestra.run.smithi170.stdout:Access: 2024-03-10 00:45:07.440037454 +0000 2024-03-10T00:45:14.179 INFO:teuthology.orchestra.run.smithi170.stdout:Modify: 2024-03-10 00:44:45.986531088 +0000 2024-03-10T00:45:14.179 INFO:teuthology.orchestra.run.smithi170.stdout:Change: 2024-03-10 00:44:45.986531088 +0000 2024-03-10T00:45:14.180 INFO:teuthology.orchestra.run.smithi170.stdout: Birth: 2024-03-10 00:44:45.986531088 +0000 2024-03-10T00:45:14.180 DEBUG:teuthology.orchestra.run.smithi170:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-03-10T00:45:14.245 INFO:teuthology.orchestra.run.smithi170.stderr:1+0 records in 2024-03-10T00:45:14.245 INFO:teuthology.orchestra.run.smithi170.stderr:1+0 records out 2024-03-10T00:45:14.245 INFO:teuthology.orchestra.run.smithi170.stderr:512 bytes copied, 0.000102672 s, 5.0 MB/s 2024-03-10T00:45:14.247 DEBUG:teuthology.orchestra.run.smithi170:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-03-10T00:45:14.303 DEBUG:teuthology.orchestra.run.smithi170:> grep '^nvme_loop' /proc/modules || sudo modprobe nvme_loop && sudo mkdir -p /sys/kernel/config/nvmet/hosts/hostnqn && sudo mkdir -p /sys/kernel/config/nvmet/ports/1 && echo loop | sudo tee /sys/kernel/config/nvmet/ports/1/addr_trtype 2024-03-10T00:45:14.449 INFO:teuthology.orchestra.run.smithi170.stdout:loop 2024-03-10T00:45:14.451 INFO:tasks.nvme_loop:Connecting nvme_loop smithi170:/dev/vg_nvme/lv_1... 2024-03-10T00:45:14.451 DEBUG:teuthology.orchestra.run.smithi170:> sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_1 && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_1/attr_allow_any_host && sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_1/namespaces/1 && echo -n /dev/vg_nvme/lv_1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_1/namespaces/1/device_path && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_1/namespaces/1/enable && sudo ln -s /sys/kernel/config/nvmet/subsystems/lv_1 /sys/kernel/config/nvmet/ports/1/subsystems/lv_1 && sudo nvme connect -t loop -n lv_1 -q hostnqn 2024-03-10T00:45:14.489 INFO:teuthology.orchestra.run.smithi170.stdout:1 2024-03-10T00:45:14.526 INFO:teuthology.orchestra.run.smithi170.stdout:/dev/vg_nvme/lv_11 2024-03-10T00:45:14.563 INFO:tasks.nvme_loop:Connecting nvme_loop smithi170:/dev/vg_nvme/lv_2... 2024-03-10T00:45:14.563 DEBUG:teuthology.orchestra.run.smithi170:> sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_2 && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_2/attr_allow_any_host && sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_2/namespaces/1 && echo -n /dev/vg_nvme/lv_2 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_2/namespaces/1/device_path && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_2/namespaces/1/enable && sudo ln -s /sys/kernel/config/nvmet/subsystems/lv_2 /sys/kernel/config/nvmet/ports/1/subsystems/lv_2 && sudo nvme connect -t loop -n lv_2 -q hostnqn 2024-03-10T00:45:14.641 INFO:teuthology.orchestra.run.smithi170.stdout:1 2024-03-10T00:45:14.679 INFO:teuthology.orchestra.run.smithi170.stdout:/dev/vg_nvme/lv_21 2024-03-10T00:45:14.714 INFO:tasks.nvme_loop:Connecting nvme_loop smithi170:/dev/vg_nvme/lv_3... 2024-03-10T00:45:14.715 DEBUG:teuthology.orchestra.run.smithi170:> sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_3 && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_3/attr_allow_any_host && sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_3/namespaces/1 && echo -n /dev/vg_nvme/lv_3 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_3/namespaces/1/device_path && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_3/namespaces/1/enable && sudo ln -s /sys/kernel/config/nvmet/subsystems/lv_3 /sys/kernel/config/nvmet/ports/1/subsystems/lv_3 && sudo nvme connect -t loop -n lv_3 -q hostnqn 2024-03-10T00:45:14.797 INFO:teuthology.orchestra.run.smithi170.stdout:1 2024-03-10T00:45:14.836 INFO:teuthology.orchestra.run.smithi170.stdout:/dev/vg_nvme/lv_31 2024-03-10T00:45:14.865 INFO:tasks.nvme_loop:Connecting nvme_loop smithi170:/dev/vg_nvme/lv_4... 2024-03-10T00:45:14.865 DEBUG:teuthology.orchestra.run.smithi170:> sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_4 && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_4/attr_allow_any_host && sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_4/namespaces/1 && echo -n /dev/vg_nvme/lv_4 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_4/namespaces/1/device_path && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_4/namespaces/1/enable && sudo ln -s /sys/kernel/config/nvmet/subsystems/lv_4 /sys/kernel/config/nvmet/ports/1/subsystems/lv_4 && sudo nvme connect -t loop -n lv_4 -q hostnqn 2024-03-10T00:45:14.944 INFO:teuthology.orchestra.run.smithi170.stdout:1 2024-03-10T00:45:14.983 INFO:teuthology.orchestra.run.smithi170.stdout:/dev/vg_nvme/lv_41 2024-03-10T00:45:15.012 DEBUG:teuthology.orchestra.run.smithi170:> set -ex 2024-03-10T00:45:15.012 DEBUG:teuthology.orchestra.run.smithi170:> dd if=/scratch_devs of=/dev/stdout 2024-03-10T00:45:15.074 DEBUG:teuthology.orchestra.run.smithi170:> sudo nvme list 2024-03-10T00:45:15.141 INFO:teuthology.orchestra.run.smithi170.stdout:Node SN Model Namespace Usage Format FW Rev 2024-03-10T00:45:15.142 INFO:teuthology.orchestra.run.smithi170.stdout:---------------- -------------------- ---------------------------------------- --------- -------------------------- ---------------- -------- 2024-03-10T00:45:15.142 INFO:teuthology.orchestra.run.smithi170.stdout:/dev/nvme0n1 PHMB7476002R480DGN INTEL SSDPED1D480GA 1 480.10 GB / 480.10 GB 512 B + 0 B E2010325 2024-03-10T00:45:15.142 INFO:teuthology.orchestra.run.smithi170.stdout:/dev/nvme1n1 0bec78b1e11737015da3 Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 5.14.0-4 2024-03-10T00:45:15.142 INFO:teuthology.orchestra.run.smithi170.stdout:/dev/nvme2n1 7f67e114c7bb103ae901 Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 5.14.0-4 2024-03-10T00:45:15.142 INFO:teuthology.orchestra.run.smithi170.stdout:/dev/nvme3n1 e5e9296aa0892d5aa3a2 Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 5.14.0-4 2024-03-10T00:45:15.142 INFO:teuthology.orchestra.run.smithi170.stdout:/dev/nvme4n1 23d280e791f50e30f40c Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 5.14.0-4 2024-03-10T00:45:15.144 INFO:tasks.nvme_loop:new_devs ['/dev/nvme1n1', '/dev/nvme2n1', '/dev/nvme3n1', '/dev/nvme4n1'] 2024-03-10T00:45:15.144 DEBUG:teuthology.orchestra.run.smithi170:> set -ex 2024-03-10T00:45:15.144 DEBUG:teuthology.orchestra.run.smithi170:> sudo dd of=/scratch_devs 2024-03-10T00:45:15.211 INFO:teuthology.run_tasks:Running task cephadm... 2024-03-10T00:45:15.327 INFO:tasks.cephadm:Config: {'conf': {'mgr': {'debug mgr': 20, 'debug ms': 1, 'mgr/cephadm/use_agent': True}, 'global': {'mon election default strategy': 3}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'debug ms': 1, 'debug osd': 20, 'osd shutdown pgref assert': True}}, 'flavor': 'default', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'MON_DOWN'], 'sha1': '831bc188fa69f143bbba69ac76119e784500e8e3'} 2024-03-10T00:45:15.327 INFO:tasks.cephadm:Cluster image is quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:831bc188fa69f143bbba69ac76119e784500e8e3 2024-03-10T00:45:15.328 INFO:tasks.cephadm:Cluster fsid is 75c37498-de77-11ee-95c7-87774f69a715 2024-03-10T00:45:15.328 INFO:tasks.cephadm:Choosing monitor IPs and ports... 2024-03-10T00:45:15.328 INFO:tasks.cephadm:Monitor IPs: {'mon.a': '172.21.15.106', 'mon.c': '[v2:172.21.15.106:3301,v1:172.21.15.106:6790]', 'mon.b': '172.21.15.170'} 2024-03-10T00:45:15.328 INFO:tasks.cephadm:First mon is mon.a on smithi106 2024-03-10T00:45:15.328 INFO:tasks.cephadm:First mgr is y 2024-03-10T00:45:15.328 INFO:tasks.cephadm:Normalizing hostnames... 2024-03-10T00:45:15.329 DEBUG:teuthology.orchestra.run.smithi106:> sudo hostname $(hostname -s) 2024-03-10T00:45:15.361 DEBUG:teuthology.orchestra.run.smithi170:> sudo hostname $(hostname -s) 2024-03-10T00:45:15.389 INFO:tasks.cephadm:Downloading "compiled" cephadm from cachra 2024-03-10T00:45:15.389 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=831bc188fa69f143bbba69ac76119e784500e8e3 2024-03-10T00:45:15.617 INFO:tasks.cephadm:builder_project result: [{'status': 'ready', 'sha1': '831bc188fa69f143bbba69ac76119e784500e8e3', 'extra': {'build_url': 'https://jenkins.ceph.com/job/ceph-dev-new-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=centos9,DIST=centos9,MACHINE_SIZE=gigantic/77345/', 'root_build_cause': 'SCMTRIGGER', 'version': '19.0.0-1875-g831bc188', 'node_name': '172.21.2.1+braggi01', 'job_name': 'ceph-dev-new-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=centos9,DIST=centos9,MACHINE_SIZE=gigantic', 'package_manager_version': '19.0.0-1875.g831bc188'}, 'url': 'https://4.chacra.ceph.com/r/ceph/wip-yuri4-testing-2024-03-05-0854/831bc188fa69f143bbba69ac76119e784500e8e3/centos/9/flavors/default/', 'distro_codename': None, 'modified': '2024-03-06 02:32:08.274555', 'distro_version': '9', 'project': 'ceph', 'flavor': 'default', 'ref': 'wip-yuri4-testing-2024-03-05-0854', 'chacra_url': 'https://4.chacra.ceph.com/repos/ceph/wip-yuri4-testing-2024-03-05-0854/831bc188fa69f143bbba69ac76119e784500e8e3/centos/9/flavors/default/', 'archs': ['source', 'x86_64'], 'distro': 'centos'}] 2024-03-10T00:45:15.755 INFO:tasks.util.chacra:got chacra host 4.chacra.ceph.com, ref wip-yuri4-testing-2024-03-05-0854, sha1 831bc188fa69f143bbba69ac76119e784500e8e3 from https://shaman.ceph.com/api/search/?project=ceph&distros=centos%2F9%2Fx86_64&flavor=default&sha1=831bc188fa69f143bbba69ac76119e784500e8e3 2024-03-10T00:45:15.756 INFO:tasks.cephadm:Discovered cachra url: https://4.chacra.ceph.com/binaries/ceph/wip-yuri4-testing-2024-03-05-0854/831bc188fa69f143bbba69ac76119e784500e8e3/centos/9/x86_64/flavors/default/cephadm 2024-03-10T00:45:15.756 DEBUG:teuthology.orchestra.run.smithi106:> curl --silent -L https://4.chacra.ceph.com/binaries/ceph/wip-yuri4-testing-2024-03-05-0854/831bc188fa69f143bbba69ac76119e784500e8e3/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-03-10T00:45:16.208 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r--. 1 ubuntu ubuntu 767443 Mar 10 00:45 /home/ubuntu/cephtest/cephadm 2024-03-10T00:45:16.208 DEBUG:teuthology.orchestra.run.smithi170:> curl --silent -L https://4.chacra.ceph.com/binaries/ceph/wip-yuri4-testing-2024-03-05-0854/831bc188fa69f143bbba69ac76119e784500e8e3/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-03-10T00:45:16.780 INFO:teuthology.orchestra.run.smithi170.stdout:-rw-r--r--. 1 ubuntu ubuntu 767443 Mar 10 00:45 /home/ubuntu/cephtest/cephadm 2024-03-10T00:45:16.780 DEBUG:teuthology.orchestra.run.smithi106:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-03-10T00:45:16.799 DEBUG:teuthology.orchestra.run.smithi170:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-03-10T00:45:16.827 INFO:tasks.cephadm:Pulling image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:831bc188fa69f143bbba69ac76119e784500e8e3 on all hosts... 2024-03-10T00:45:16.827 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:831bc188fa69f143bbba69ac76119e784500e8e3 pull 2024-03-10T00:45:16.841 DEBUG:teuthology.orchestra.run.smithi170:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:831bc188fa69f143bbba69ac76119e784500e8e3 pull 2024-03-10T00:45:17.646 INFO:teuthology.orchestra.run.smithi170.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:831bc188fa69f143bbba69ac76119e784500e8e3... 2024-03-10T00:45:17.671 INFO:teuthology.orchestra.run.smithi106.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:831bc188fa69f143bbba69ac76119e784500e8e3... 2024-03-10T00:45:47.582 INFO:teuthology.orchestra.run.smithi106.stdout:{ 2024-03-10T00:45:47.582 INFO:teuthology.orchestra.run.smithi106.stdout: "ceph_version": "ceph version 19.0.0-1875-g831bc188 (831bc188fa69f143bbba69ac76119e784500e8e3) squid (dev)", 2024-03-10T00:45:47.582 INFO:teuthology.orchestra.run.smithi106.stdout: "image_id": "01a4f6a6bb3e92ab9ab39a2f835684ea9b10e6ec9fb81ac08ebea4165583a538", 2024-03-10T00:45:47.582 INFO:teuthology.orchestra.run.smithi106.stdout: "repo_digests": [ 2024-03-10T00:45:47.582 INFO:teuthology.orchestra.run.smithi106.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bc4a24690ce5d7255d490e21058d8c114b55e1b93a3d8345f90b6700ad29dada" 2024-03-10T00:45:47.583 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-03-10T00:45:47.583 INFO:teuthology.orchestra.run.smithi106.stdout:} 2024-03-10T00:45:48.930 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2024-03-10T00:45:48.930 INFO:teuthology.orchestra.run.smithi170.stdout: "ceph_version": "ceph version 19.0.0-1875-g831bc188 (831bc188fa69f143bbba69ac76119e784500e8e3) squid (dev)", 2024-03-10T00:45:48.930 INFO:teuthology.orchestra.run.smithi170.stdout: "image_id": "01a4f6a6bb3e92ab9ab39a2f835684ea9b10e6ec9fb81ac08ebea4165583a538", 2024-03-10T00:45:48.930 INFO:teuthology.orchestra.run.smithi170.stdout: "repo_digests": [ 2024-03-10T00:45:48.930 INFO:teuthology.orchestra.run.smithi170.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bc4a24690ce5d7255d490e21058d8c114b55e1b93a3d8345f90b6700ad29dada" 2024-03-10T00:45:48.930 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2024-03-10T00:45:48.930 INFO:teuthology.orchestra.run.smithi170.stdout:} 2024-03-10T00:45:48.954 DEBUG:teuthology.orchestra.run.smithi106:> sudo mkdir -p /etc/ceph 2024-03-10T00:45:48.988 DEBUG:teuthology.orchestra.run.smithi170:> sudo mkdir -p /etc/ceph 2024-03-10T00:45:49.018 DEBUG:teuthology.orchestra.run.smithi106:> sudo chmod 777 /etc/ceph 2024-03-10T00:45:49.059 DEBUG:teuthology.orchestra.run.smithi170:> sudo chmod 777 /etc/ceph 2024-03-10T00:45:49.086 INFO:tasks.cephadm:Writing seed config... 2024-03-10T00:45:49.087 INFO:tasks.cephadm: override: [mgr] debug mgr = 20 2024-03-10T00:45:49.087 INFO:tasks.cephadm: override: [mgr] debug ms = 1 2024-03-10T00:45:49.087 INFO:tasks.cephadm: override: [mgr] mgr/cephadm/use_agent = True 2024-03-10T00:45:49.087 INFO:tasks.cephadm: override: [global] mon election default strategy = 3 2024-03-10T00:45:49.087 INFO:tasks.cephadm: override: [mon] debug mon = 20 2024-03-10T00:45:49.087 INFO:tasks.cephadm: override: [mon] debug ms = 1 2024-03-10T00:45:49.087 INFO:tasks.cephadm: override: [mon] debug paxos = 20 2024-03-10T00:45:49.088 INFO:tasks.cephadm: override: [osd] debug ms = 1 2024-03-10T00:45:49.088 INFO:tasks.cephadm: override: [osd] debug osd = 20 2024-03-10T00:45:49.088 INFO:tasks.cephadm: override: [osd] osd shutdown pgref assert = True 2024-03-10T00:45:49.088 DEBUG:teuthology.orchestra.run.smithi106:> set -ex 2024-03-10T00:45:49.088 DEBUG:teuthology.orchestra.run.smithi106:> dd of=/home/ubuntu/cephtest/seed.ceph.conf 2024-03-10T00:45:49.116 DEBUG:tasks.cephadm:Final config: [global] # make logging friendly to teuthology log_to_file = true log_to_stderr = false log to journald = false mon cluster log to file = true mon cluster log file level = debug mon clock drift allowed = 1.000 # replicate across OSDs, not hosts osd crush chooseleaf type = 0 #osd pool default size = 2 osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd # enable some debugging auth debug = true ms die on old message = true ms die on bug = true debug asserts on shutdown = true # adjust warnings mon max pg per osd = 10000# >= luminous mon pg warn max object skew = 0 mon osd allow primary affinity = true mon osd allow pg remap = true mon warn on legacy crush tunables = false mon warn on crush straw calc version zero = false mon warn on no sortbitwise = false mon warn on osd down out interval zero = false mon warn on too few osds = false mon_warn_on_pool_pg_num_not_power_of_two = false # disable pg_autoscaler by default for new pools osd_pool_default_pg_autoscale_mode = off # tests delete pools mon allow pool delete = true fsid = 75c37498-de77-11ee-95c7-87774f69a715 mon election default strategy = 3 [osd] osd scrub load threshold = 5.0 osd scrub max interval = 600 osd mclock profile = high_recovery_ops osd recover clone overlap = true osd recovery max chunk = 1048576 osd deep scrub update digest min age = 30 osd map max advance = 10 osd memory target autotune = true # debugging osd debug shutdown = true osd debug op order = true osd debug verify stray on activate = true osd debug pg log writeout = true osd debug verify cached snaps = true osd debug verify missing on start = true osd debug misdirected ops = true osd op queue = debug_random osd op queue cut off = debug_random osd shutdown pgref assert = True bdev debug aio = true osd sloppy crc = true debug ms = 1 debug osd = 20 [mgr] mon reweight min pgs per osd = 4 mon reweight min bytes per osd = 10 mgr/telemetry/nag = false debug mgr = 20 debug ms = 1 mgr/cephadm/use_agent = True [mon] mon data avail warn = 5 mon mgr mkfs grace = 240 mon reweight min pgs per osd = 4 mon osd reporter subtree level = osd mon osd prime pg temp = true mon reweight min bytes per osd = 10 # rotate auth tickets quickly to exercise renewal paths auth mon ticket ttl = 660# 11m auth service ticket ttl = 240# 4m # don't complain about global id reclaim mon_warn_on_insecure_global_id_reclaim = false mon_warn_on_insecure_global_id_reclaim_allowed = false debug mon = 20 debug ms = 1 debug paxos = 20 [client.rgw] rgw cache enabled = true rgw enable ops log = true rgw enable usage log = true 2024-03-10T00:45:49.116 DEBUG:teuthology.orchestra.run.smithi106:mon.a> sudo journalctl -f -n 0 -u ceph-75c37498-de77-11ee-95c7-87774f69a715@mon.a.service 2024-03-10T00:45:49.159 DEBUG:teuthology.orchestra.run.smithi106:mgr.y> sudo journalctl -f -n 0 -u ceph-75c37498-de77-11ee-95c7-87774f69a715@mgr.y.service 2024-03-10T00:45:49.202 INFO:tasks.cephadm:Bootstrapping... 2024-03-10T00:45:49.203 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:831bc188fa69f143bbba69ac76119e784500e8e3 -v bootstrap --fsid 75c37498-de77-11ee-95c7-87774f69a715 --config /home/ubuntu/cephtest/seed.ceph.conf --output-config /etc/ceph/ceph.conf --output-keyring /etc/ceph/ceph.client.admin.keyring --output-pub-ssh-key /home/ubuntu/cephtest/ceph.pub --mon-id a --mgr-id y --orphan-initial-daemons --skip-monitoring-stack --mon-ip 172.21.15.106 --skip-admin-label && sudo chmod +r /etc/ceph/ceph.client.admin.keyring 2024-03-10T00:45:49.395 INFO:teuthology.orchestra.run.smithi106.stdout:-------------------------------------------------------------------------------- 2024-03-10T00:45:49.395 INFO:teuthology.orchestra.run.smithi106.stdout:cephadm ['--image', 'quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:831bc188fa69f143bbba69ac76119e784500e8e3', '-v', 'bootstrap', '--fsid', '75c37498-de77-11ee-95c7-87774f69a715', '--config', '/home/ubuntu/cephtest/seed.ceph.conf', '--output-config', '/etc/ceph/ceph.conf', '--output-keyring', '/etc/ceph/ceph.client.admin.keyring', '--output-pub-ssh-key', '/home/ubuntu/cephtest/ceph.pub', '--mon-id', 'a', '--mgr-id', 'y', '--orphan-initial-daemons', '--skip-monitoring-stack', '--mon-ip', '172.21.15.106', '--skip-admin-label'] 2024-03-10T00:45:49.420 INFO:teuthology.orchestra.run.smithi106.stdout:/bin/podman: stdout 4.9.3-dev 2024-03-10T00:45:49.420 INFO:teuthology.orchestra.run.smithi106.stderr:Specifying an fsid for your cluster offers no advantages and may increase the likelihood of fsid conflicts. 2024-03-10T00:45:49.420 INFO:teuthology.orchestra.run.smithi106.stdout:Verifying podman|docker is present... 2024-03-10T00:45:49.443 INFO:teuthology.orchestra.run.smithi106.stdout:/bin/podman: stdout 4.9.3-dev 2024-03-10T00:45:49.443 INFO:teuthology.orchestra.run.smithi106.stdout:Verifying lvm2 is present... 2024-03-10T00:45:49.443 INFO:teuthology.orchestra.run.smithi106.stdout:Verifying time synchronization is in place... 2024-03-10T00:45:49.450 INFO:teuthology.orchestra.run.smithi106.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-03-10T00:45:49.451 INFO:teuthology.orchestra.run.smithi106.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-03-10T00:45:49.456 INFO:teuthology.orchestra.run.smithi106.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-03-10T00:45:49.456 INFO:teuthology.orchestra.run.smithi106.stdout:systemctl: stdout inactive 2024-03-10T00:45:49.463 INFO:teuthology.orchestra.run.smithi106.stdout:systemctl: stdout enabled 2024-03-10T00:45:49.468 INFO:teuthology.orchestra.run.smithi106.stdout:systemctl: stdout active 2024-03-10T00:45:49.468 INFO:teuthology.orchestra.run.smithi106.stdout:Unit chronyd.service is enabled and running 2024-03-10T00:45:49.468 INFO:teuthology.orchestra.run.smithi106.stdout:Repeating the final host check... 2024-03-10T00:45:49.492 INFO:teuthology.orchestra.run.smithi106.stdout:/bin/podman: stdout 4.9.3-dev 2024-03-10T00:45:49.492 INFO:teuthology.orchestra.run.smithi106.stdout:podman (/bin/podman) version 4.9.3 is present 2024-03-10T00:45:49.492 INFO:teuthology.orchestra.run.smithi106.stdout:systemctl is present 2024-03-10T00:45:49.492 INFO:teuthology.orchestra.run.smithi106.stdout:lvcreate is present 2024-03-10T00:45:49.498 INFO:teuthology.orchestra.run.smithi106.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-03-10T00:45:49.498 INFO:teuthology.orchestra.run.smithi106.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-03-10T00:45:49.503 INFO:teuthology.orchestra.run.smithi106.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-03-10T00:45:49.504 INFO:teuthology.orchestra.run.smithi106.stdout:systemctl: stdout inactive 2024-03-10T00:45:49.510 INFO:teuthology.orchestra.run.smithi106.stdout:systemctl: stdout enabled 2024-03-10T00:45:49.516 INFO:teuthology.orchestra.run.smithi106.stdout:systemctl: stdout active 2024-03-10T00:45:49.516 INFO:teuthology.orchestra.run.smithi106.stdout:Unit chronyd.service is enabled and running 2024-03-10T00:45:49.516 INFO:teuthology.orchestra.run.smithi106.stdout:Host looks OK 2024-03-10T00:45:49.516 INFO:teuthology.orchestra.run.smithi106.stdout:Cluster fsid: 75c37498-de77-11ee-95c7-87774f69a715 2024-03-10T00:45:49.516 INFO:teuthology.orchestra.run.smithi106.stdout:Acquiring lock 139886318216576 on /run/cephadm/75c37498-de77-11ee-95c7-87774f69a715.lock 2024-03-10T00:45:49.516 INFO:teuthology.orchestra.run.smithi106.stdout:Lock 139886318216576 acquired on /run/cephadm/75c37498-de77-11ee-95c7-87774f69a715.lock 2024-03-10T00:45:49.517 INFO:teuthology.orchestra.run.smithi106.stdout:Verifying IP 172.21.15.106 port 3300 ... 2024-03-10T00:45:49.517 INFO:teuthology.orchestra.run.smithi106.stdout:Verifying IP 172.21.15.106 port 6789 ... 2024-03-10T00:45:49.517 INFO:teuthology.orchestra.run.smithi106.stdout:Base mon IP(s) is [172.21.15.106:3300, 172.21.15.106:6789], mon addrv is [v2:172.21.15.106:3300,v1:172.21.15.106:6789] 2024-03-10T00:45:49.520 INFO:teuthology.orchestra.run.smithi106.stdout:/sbin/ip: stdout default via 172.21.15.254 dev enp3s0f1 proto dhcp src 172.21.15.106 metric 100 2024-03-10T00:45:49.520 INFO:teuthology.orchestra.run.smithi106.stdout:/sbin/ip: stdout 172.21.0.0/20 dev enp3s0f1 proto kernel scope link src 172.21.15.106 metric 100 2024-03-10T00:45:49.522 INFO:teuthology.orchestra.run.smithi106.stdout:/sbin/ip: stdout ::1 dev lo proto kernel metric 256 pref medium 2024-03-10T00:45:49.523 INFO:teuthology.orchestra.run.smithi106.stdout:/sbin/ip: stdout fe80::/64 dev enp3s0f1 proto kernel metric 256 pref medium 2024-03-10T00:45:49.523 INFO:teuthology.orchestra.run.smithi106.stdout:/sbin/ip: stdout default via fe80::327c:5e00:6487:71e0 dev enp3s0f1 proto ra metric 1024 expires 1788sec hoplimit 64 pref medium 2024-03-10T00:45:49.525 INFO:teuthology.orchestra.run.smithi106.stdout:/sbin/ip: stdout 1: lo: mtu 65536 state UNKNOWN qlen 1000 2024-03-10T00:45:49.525 INFO:teuthology.orchestra.run.smithi106.stdout:/sbin/ip: stdout inet6 ::1/128 scope host 2024-03-10T00:45:49.525 INFO:teuthology.orchestra.run.smithi106.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-03-10T00:45:49.525 INFO:teuthology.orchestra.run.smithi106.stdout:/sbin/ip: stdout 5: enp3s0f1: mtu 1500 state UP qlen 1000 2024-03-10T00:45:49.525 INFO:teuthology.orchestra.run.smithi106.stdout:/sbin/ip: stdout inet6 fe80::ec4:7aff:fe8f:ce6d/64 scope link 2024-03-10T00:45:49.525 INFO:teuthology.orchestra.run.smithi106.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-03-10T00:45:49.526 INFO:teuthology.orchestra.run.smithi106.stdout:Mon IP `172.21.15.106` is in CIDR network `172.21.0.0/20` 2024-03-10T00:45:49.526 INFO:teuthology.orchestra.run.smithi106.stdout:Mon IP `172.21.15.106` is in CIDR network `172.21.0.0/20` 2024-03-10T00:45:49.526 INFO:teuthology.orchestra.run.smithi106.stdout:Inferred mon public CIDR from local network configuration ['172.21.0.0/20', '172.21.0.0/20'] 2024-03-10T00:45:49.526 INFO:teuthology.orchestra.run.smithi106.stdout:Internal network (--cluster-network) has not been provided, OSD replication will default to the public_network 2024-03-10T00:45:49.527 INFO:teuthology.orchestra.run.smithi106.stdout:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:831bc188fa69f143bbba69ac76119e784500e8e3... 2024-03-10T00:45:49.984 INFO:teuthology.orchestra.run.smithi106.stdout:/bin/podman: stdout 01a4f6a6bb3e92ab9ab39a2f835684ea9b10e6ec9fb81ac08ebea4165583a538 2024-03-10T00:45:49.984 INFO:teuthology.orchestra.run.smithi106.stdout:/bin/podman: stderr Trying to pull quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:831bc188fa69f143bbba69ac76119e784500e8e3... 2024-03-10T00:45:49.984 INFO:teuthology.orchestra.run.smithi106.stdout:/bin/podman: stderr Getting image source signatures 2024-03-10T00:45:49.984 INFO:teuthology.orchestra.run.smithi106.stdout:/bin/podman: stderr Copying blob sha256:5230e21a7b1561343f4233b6671f80ad8914d8a2054fff55ddda8b9974582e35 2024-03-10T00:45:49.984 INFO:teuthology.orchestra.run.smithi106.stdout:/bin/podman: stderr Copying blob sha256:7feca07754707458c3945cf0062cf4dabc512f6d90fe1a9a1370b362b6011124 2024-03-10T00:45:49.984 INFO:teuthology.orchestra.run.smithi106.stdout:/bin/podman: stderr Copying config sha256:01a4f6a6bb3e92ab9ab39a2f835684ea9b10e6ec9fb81ac08ebea4165583a538 2024-03-10T00:45:49.984 INFO:teuthology.orchestra.run.smithi106.stdout:/bin/podman: stderr Writing manifest to image destination 2024-03-10T00:45:50.904 INFO:teuthology.orchestra.run.smithi106.stdout:ceph: stdout ceph version 19.0.0-1875-g831bc188 (831bc188fa69f143bbba69ac76119e784500e8e3) squid (dev) 2024-03-10T00:45:50.905 INFO:teuthology.orchestra.run.smithi106.stdout:Ceph version: ceph version 19.0.0-1875-g831bc188 (831bc188fa69f143bbba69ac76119e784500e8e3) squid (dev) 2024-03-10T00:45:50.905 INFO:teuthology.orchestra.run.smithi106.stdout:Extracting ceph user uid/gid from container image... 2024-03-10T00:45:51.777 INFO:teuthology.orchestra.run.smithi106.stdout:stat: stdout 167 167 2024-03-10T00:45:51.777 INFO:teuthology.orchestra.run.smithi106.stdout:Creating initial keys... 2024-03-10T00:45:52.798 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph-authtool: stdout AQDAAu1lA9zFBhAAQEdEachKrb3wL0AKHJnYAw== 2024-03-10T00:45:53.678 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph-authtool: stdout AQDBAu1lXXEdBRAAq8GoiLte9AsLPxQvjWlzmQ== 2024-03-10T00:45:54.710 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph-authtool: stdout AQDCAu1lhaDVDxAAXxmIVd4AsIAAn2K33WDvog== 2024-03-10T00:45:54.711 INFO:teuthology.orchestra.run.smithi106.stdout:Creating initial monmap... 2024-03-10T00:45:55.605 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: monmap file /tmp/monmap 2024-03-10T00:45:55.605 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/monmaptool: stdout setting min_mon_release = quincy 2024-03-10T00:45:55.605 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: set fsid to 75c37498-de77-11ee-95c7-87774f69a715 2024-03-10T00:45:55.605 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-03-10T00:45:55.606 INFO:teuthology.orchestra.run.smithi106.stdout:monmaptool for a [v2:172.21.15.106:3300,v1:172.21.15.106:6789] on /usr/bin/monmaptool: monmap file /tmp/monmap 2024-03-10T00:45:55.606 INFO:teuthology.orchestra.run.smithi106.stdout:setting min_mon_release = quincy 2024-03-10T00:45:55.606 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/monmaptool: set fsid to 75c37498-de77-11ee-95c7-87774f69a715 2024-03-10T00:45:55.606 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-03-10T00:45:55.606 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-03-10T00:45:55.606 INFO:teuthology.orchestra.run.smithi106.stdout:Creating mon... 2024-03-10T00:45:58.046 INFO:teuthology.orchestra.run.smithi106.stdout:create mon.a on 2024-03-10T00:45:58.793 INFO:teuthology.orchestra.run.smithi106.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /etc/systemd/system/ceph.target. 2024-03-10T00:45:59.181 INFO:teuthology.orchestra.run.smithi106.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph-75c37498-de77-11ee-95c7-87774f69a715.target → /etc/systemd/system/ceph-75c37498-de77-11ee-95c7-87774f69a715.target. 2024-03-10T00:45:59.181 INFO:teuthology.orchestra.run.smithi106.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph.target.wants/ceph-75c37498-de77-11ee-95c7-87774f69a715.target → /etc/systemd/system/ceph-75c37498-de77-11ee-95c7-87774f69a715.target. 2024-03-10T00:46:00.037 INFO:teuthology.orchestra.run.smithi106.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-75c37498-de77-11ee-95c7-87774f69a715@mon.a 2024-03-10T00:46:00.037 INFO:teuthology.orchestra.run.smithi106.stdout:systemctl: stderr Failed to reset failed state of unit ceph-75c37498-de77-11ee-95c7-87774f69a715@mon.a.service: Unit ceph-75c37498-de77-11ee-95c7-87774f69a715@mon.a.service not loaded. 2024-03-10T00:46:00.334 INFO:teuthology.orchestra.run.smithi106.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-75c37498-de77-11ee-95c7-87774f69a715.target.wants/ceph-75c37498-de77-11ee-95c7-87774f69a715@mon.a.service → /etc/systemd/system/ceph-75c37498-de77-11ee-95c7-87774f69a715@.service. 2024-03-10T00:46:00.839 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:46:00 smithi106 podman[25208]: 2024-03-10 00:46:00.48118615 +0000 UTC m=+0.017750746 image pull 01a4f6a6bb3e92ab9ab39a2f835684ea9b10e6ec9fb81ac08ebea4165583a538 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:831bc188fa69f143bbba69ac76119e784500e8e3 2024-03-10T00:46:01.839 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:46:01 smithi106 podman[25208]: 2024-03-10 00:46:01.485115789 +0000 UTC m=+1.021680370 container create 579895f559457b338dfe3df2a825c1d5534d3fae7ac7e95253450eb654ab0557 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:831bc188fa69f143bbba69ac76119e784500e8e3, name=ceph-75c37498-de77-11ee-95c7-87774f69a715-mon-a, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, RELEASE=wip-yuri4-testing-2024-03-05-0854-831bc18, org.label-schema.license=GPLv2, GIT_CLEAN=True, ceph=True, org.label-schema.build-date=20240222, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-10T00:46:03.589 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:46:03 smithi106 podman[25208]: 2024-03-10 00:46:03.331832148 +0000 UTC m=+2.868396749 container init 579895f559457b338dfe3df2a825c1d5534d3fae7ac7e95253450eb654ab0557 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:831bc188fa69f143bbba69ac76119e784500e8e3, name=ceph-75c37498-de77-11ee-95c7-87774f69a715-mon-a, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.build-date=20240222, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-yuri4-testing-2024-03-05-0854-831bc18) 2024-03-10T00:46:03.589 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:46:03 smithi106 podman[25208]: 2024-03-10 00:46:03.338322577 +0000 UTC m=+2.874887162 container start 579895f559457b338dfe3df2a825c1d5534d3fae7ac7e95253450eb654ab0557 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:831bc188fa69f143bbba69ac76119e784500e8e3, name=ceph-75c37498-de77-11ee-95c7-87774f69a715-mon-a, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_CLEAN=True, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240222, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, maintainer=Guillaume Abrioux , RELEASE=wip-yuri4-testing-2024-03-05-0854-831bc18, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5) 2024-03-10T00:46:03.590 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:46:03 smithi106 ceph-mon[25246]: mkfs 75c37498-de77-11ee-95c7-87774f69a715 2024-03-10T00:46:03.590 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:46:03 smithi106 ceph-mon[25246]: mon.a is new leader, mons a in quorum (ranks 0) 2024-03-10T00:46:04.169 INFO:teuthology.orchestra.run.smithi106.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-03-10T00:46:04.169 INFO:teuthology.orchestra.run.smithi106.stdout:systemctl: stdout disabled 2024-03-10T00:46:04.177 INFO:teuthology.orchestra.run.smithi106.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-03-10T00:46:04.177 INFO:teuthology.orchestra.run.smithi106.stdout:systemctl: stdout inactive 2024-03-10T00:46:04.177 INFO:teuthology.orchestra.run.smithi106.stdout:firewalld.service is not enabled 2024-03-10T00:46:04.177 INFO:teuthology.orchestra.run.smithi106.stdout:Not possible to enable service . firewalld.service is not available 2024-03-10T00:46:04.178 INFO:teuthology.orchestra.run.smithi106.stdout:Waiting for mon to start... 2024-03-10T00:46:04.178 INFO:teuthology.orchestra.run.smithi106.stdout:Waiting for mon... 2024-03-10T00:46:04.398 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:46:04 smithi106 bash[25208]: 579895f559457b338dfe3df2a825c1d5534d3fae7ac7e95253450eb654ab0557 2024-03-10T00:46:04.399 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:46:04 smithi106 systemd[1]: Started Ceph mon.a for 75c37498-de77-11ee-95c7-87774f69a715. 2024-03-10T00:46:04.654 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:46:04 smithi106 ceph-mon[25246]: mon.a is new leader, mons a in quorum (ranks 0) 2024-03-10T00:46:04.654 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:46:04 smithi106 ceph-mon[25246]: monmap e1: 1 mons at {a=[v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-10T00:46:04.654 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:46:04 smithi106 ceph-mon[25246]: fsmap 2024-03-10T00:46:04.654 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:46:04 smithi106 ceph-mon[25246]: osdmap e1: 0 total, 0 up, 0 in 2024-03-10T00:46:04.654 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:46:04 smithi106 ceph-mon[25246]: mgrmap e1: no daemons active 2024-03-10T00:46:05.558 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:46:05 smithi106 ceph-mon[25246]: from='client.? 172.21.15.106:0/3136247167' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch 2024-03-10T00:46:05.562 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout cluster: 2024-03-10T00:46:05.562 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout id: 75c37498-de77-11ee-95c7-87774f69a715 2024-03-10T00:46:05.562 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout health: HEALTH_OK 2024-03-10T00:46:05.562 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout 2024-03-10T00:46:05.562 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout services: 2024-03-10T00:46:05.562 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout mon: 1 daemons, quorum a (age 1.56266s) 2024-03-10T00:46:05.562 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout mgr: no daemons active 2024-03-10T00:46:05.562 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout osd: 0 osds: 0 up, 0 in 2024-03-10T00:46:05.562 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout 2024-03-10T00:46:05.562 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout data: 2024-03-10T00:46:05.562 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout pools: 0 pools, 0 pgs 2024-03-10T00:46:05.563 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout objects: 0 objects, 0 B 2024-03-10T00:46:05.563 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout usage: 0 B used, 0 B / 0 B avail 2024-03-10T00:46:05.563 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout pgs: 2024-03-10T00:46:05.563 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout 2024-03-10T00:46:05.563 INFO:teuthology.orchestra.run.smithi106.stdout:mon is available 2024-03-10T00:46:05.563 INFO:teuthology.orchestra.run.smithi106.stdout:Assimilating anything we can from ceph.conf... 2024-03-10T00:46:06.567 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout 2024-03-10T00:46:06.567 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout [global] 2024-03-10T00:46:06.567 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout fsid = 75c37498-de77-11ee-95c7-87774f69a715 2024-03-10T00:46:06.567 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout mon_host = [v2:172.21.15.106:3300,v1:172.21.15.106:6789] 2024-03-10T00:46:06.567 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-03-10T00:46:06.567 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-03-10T00:46:06.567 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-03-10T00:46:06.568 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-03-10T00:46:06.568 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout 2024-03-10T00:46:06.568 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout [mgr] 2024-03-10T00:46:06.568 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout mgr/cephadm/use_agent = True 2024-03-10T00:46:06.568 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-03-10T00:46:06.568 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout 2024-03-10T00:46:06.568 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout [osd] 2024-03-10T00:46:06.568 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-03-10T00:46:06.568 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-03-10T00:46:06.568 INFO:teuthology.orchestra.run.smithi106.stdout:Generating new minimal ceph.conf... 2024-03-10T00:46:06.680 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:46:06 smithi106 ceph-mon[25246]: from='client.? 172.21.15.106:0/3779276185' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-03-10T00:46:06.681 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:46:06 smithi106 ceph-mon[25246]: from='client.? 172.21.15.106:0/3779276185' entity='client.admin' cmd='[{"prefix": "config assimilate-conf"}]': finished 2024-03-10T00:46:07.629 INFO:teuthology.orchestra.run.smithi106.stdout:Restarting the monitor... 2024-03-10T00:46:07.647 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:46:07 smithi106 ceph-mon[25246]: from='client.? 172.21.15.106:0/2069989479' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-10T00:46:08.007 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:46:07 smithi106 systemd[1]: Stopping Ceph mon.a for 75c37498-de77-11ee-95c7-87774f69a715... 2024-03-10T00:46:08.007 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:46:07 smithi106 ceph-75c37498-de77-11ee-95c7-87774f69a715-mon-a[25223]: 2024-03-10T00:46:07.867+0000 7f3859d7c700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-03-10T00:46:08.007 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:46:07 smithi106 ceph-75c37498-de77-11ee-95c7-87774f69a715-mon-a[25223]: 2024-03-10T00:46:07.867+0000 7f3859d7c700 -1 mon.a@0(leader) e1 *** Got Signal Terminated *** 2024-03-10T00:46:08.339 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:46:08 smithi106 podman[25583]: 2024-03-10 00:46:08.093367489 +0000 UTC m=+0.303714339 container died 579895f559457b338dfe3df2a825c1d5534d3fae7ac7e95253450eb654ab0557 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:831bc188fa69f143bbba69ac76119e784500e8e3, name=ceph-75c37498-de77-11ee-95c7-87774f69a715-mon-a, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.build-date=20240222, org.label-schema.vendor=CentOS, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, RELEASE=wip-yuri4-testing-2024-03-05-0854-831bc18, GIT_CLEAN=True) 2024-03-10T00:46:08.817 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:46:08 smithi106 podman[25583]: 2024-03-10 00:46:08.423100492 +0000 UTC m=+0.633447343 container cleanup 579895f559457b338dfe3df2a825c1d5534d3fae7ac7e95253450eb654ab0557 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:831bc188fa69f143bbba69ac76119e784500e8e3, name=ceph-75c37498-de77-11ee-95c7-87774f69a715-mon-a, CEPH_POINT_RELEASE=, RELEASE=wip-yuri4-testing-2024-03-05-0854-831bc18, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, org.label-schema.build-date=20240222, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_CLEAN=True, io.buildah.version=1.33.5) 2024-03-10T00:46:08.817 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:46:08 smithi106 bash[25583]: ceph-75c37498-de77-11ee-95c7-87774f69a715-mon-a 2024-03-10T00:46:08.817 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:46:08 smithi106 podman[25603]: 2024-03-10 00:46:08.599011452 +0000 UTC m=+0.508886233 container remove 579895f559457b338dfe3df2a825c1d5534d3fae7ac7e95253450eb654ab0557 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:831bc188fa69f143bbba69ac76119e784500e8e3, name=ceph-75c37498-de77-11ee-95c7-87774f69a715-mon-a, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, org.label-schema.vendor=CentOS, GIT_CLEAN=True, RELEASE=wip-yuri4-testing-2024-03-05-0854-831bc18, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_BRANCH=HEAD, org.label-schema.build-date=20240222, org.label-schema.license=GPLv2) 2024-03-10T00:46:09.089 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:46:08 smithi106 systemd[1]: ceph-75c37498-de77-11ee-95c7-87774f69a715@mon.a.service: Deactivated successfully. 2024-03-10T00:46:09.089 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:46:08 smithi106 systemd[1]: Stopped Ceph mon.a for 75c37498-de77-11ee-95c7-87774f69a715. 2024-03-10T00:46:09.089 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:46:08 smithi106 systemd[1]: Starting Ceph mon.a for 75c37498-de77-11ee-95c7-87774f69a715... 2024-03-10T00:46:09.089 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:46:09 smithi106 podman[25704]: 2024-03-10 00:46:09.032958425 +0000 UTC m=+0.108804649 container create 97a03d7ed61790350cb5cb41bd58fe8d1f02ce8a94e85b2c0994a6ae2a89679b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:831bc188fa69f143bbba69ac76119e784500e8e3, name=ceph-75c37498-de77-11ee-95c7-87774f69a715-mon-a, org.label-schema.build-date=20240222, RELEASE=wip-yuri4-testing-2024-03-05-0854-831bc18, ceph=True, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5) 2024-03-10T00:46:09.090 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:46:09 smithi106 podman[25704]: 2024-03-10 00:46:08.941209636 +0000 UTC m=+0.017055860 image pull 01a4f6a6bb3e92ab9ab39a2f835684ea9b10e6ec9fb81ac08ebea4165583a538 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:831bc188fa69f143bbba69ac76119e784500e8e3 2024-03-10T00:46:10.339 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:46:10 smithi106 podman[25704]: 2024-03-10 00:46:10.084403678 +0000 UTC m=+1.160249916 container init 97a03d7ed61790350cb5cb41bd58fe8d1f02ce8a94e85b2c0994a6ae2a89679b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:831bc188fa69f143bbba69ac76119e784500e8e3, name=ceph-75c37498-de77-11ee-95c7-87774f69a715-mon-a, org.label-schema.build-date=20240222, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, RELEASE=wip-yuri4-testing-2024-03-05-0854-831bc18, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, ceph=True, org.label-schema.schema-version=1.0) 2024-03-10T00:46:10.340 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:46:10 smithi106 podman[25704]: 2024-03-10 00:46:10.091281134 +0000 UTC m=+1.167127359 container start 97a03d7ed61790350cb5cb41bd58fe8d1f02ce8a94e85b2c0994a6ae2a89679b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:831bc188fa69f143bbba69ac76119e784500e8e3, name=ceph-75c37498-de77-11ee-95c7-87774f69a715-mon-a, RELEASE=wip-yuri4-testing-2024-03-05-0854-831bc18, CEPH_POINT_RELEASE=, GIT_CLEAN=True, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.build-date=20240222, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS) 2024-03-10T00:46:10.340 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:46:10 smithi106 ceph-mon[25742]: set uid:gid to 167:167 (ceph:ceph) 2024-03-10T00:46:10.340 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:46:10 smithi106 ceph-mon[25742]: ceph version 19.0.0-1875-g831bc188 (831bc188fa69f143bbba69ac76119e784500e8e3) squid (dev), process ceph-mon, pid 7 2024-03-10T00:46:10.340 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:46:10 smithi106 ceph-mon[25742]: pidfile_write: ignore empty --pid-file 2024-03-10T00:46:10.340 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:46:10 smithi106 ceph-mon[25742]: load: jerasure load: lrc 2024-03-10T00:46:10.340 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:46:10 smithi106 ceph-mon[25742]: rocksdb: RocksDB version: 7.9.2 2024-03-10T00:46:10.340 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:46:10 smithi106 ceph-mon[25742]: rocksdb: Git sha 0 2024-03-10T00:46:10.340 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:46:10 smithi106 ceph-mon[25742]: rocksdb: Compile date 2024-03-06 01:10:31 2024-03-10T00:46:10.340 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:46:10 smithi106 ceph-mon[25742]: rocksdb: DB SUMMARY 2024-03-10T00:46:10.340 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:46:10 smithi106 ceph-mon[25742]: rocksdb: DB Session ID: CTU7EGIYN8XQK7H0GSN2 2024-03-10T00:46:10.340 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:46:10 smithi106 ceph-mon[25742]: rocksdb: CURRENT file: CURRENT 2024-03-10T00:46:10.340 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:46:10 smithi106 ceph-mon[25742]: rocksdb: IDENTITY file: IDENTITY 2024-03-10T00:46:10.341 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:46:10 smithi106 ceph-mon[25742]: rocksdb: MANIFEST file: MANIFEST-000010 size: 179 Bytes 2024-03-10T00:46:10.341 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:46:10 smithi106 ceph-mon[25742]: rocksdb: SST files in /var/lib/ceph/mon/ceph-a/store.db dir, Total Num: 1, files: 000008.sst 2024-03-10T00:46:10.341 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:46:10 smithi106 ceph-mon[25742]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-a/store.db: 000009.log size: 84952 ; 2024-03-10T00:46:10.341 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:46:10 smithi106 ceph-mon[25742]: rocksdb: Options.error_if_exists: 0 2024-03-10T00:46:10.341 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:46:10 smithi106 ceph-mon[25742]: rocksdb: Options.create_if_missing: 0 2024-03-10T00:46:10.341 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:46:10 smithi106 ceph-mon[25742]: rocksdb: Options.paranoid_checks: 1 2024-03-10T00:46:10.341 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:46:10 smithi106 ceph-mon[25742]: rocksdb: Options.flush_verify_memtable_count: 1 2024-03-10T00:46:10.341 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:46:10 smithi106 ceph-mon[25742]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-03-10T00:46:10.341 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:46:10 smithi106 ceph-mon[25742]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-03-10T00:46:10.341 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:46:10 smithi106 ceph-mon[25742]: rocksdb: Options.env: 0x557855627b20 2024-03-10T00:46:10.341 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:46:10 smithi106 ceph-mon[25742]: rocksdb: Options.fs: PosixFileSystem 2024-03-10T00:46:10.341 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:46:10 smithi106 ceph-mon[25742]: rocksdb: Options.info_log: 0x5578570e58e0 2024-03-10T00:46:10.341 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:46:10 smithi106 ceph-mon[25742]: rocksdb: Options.max_file_opening_threads: 16 2024-03-10T00:46:10.342 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:46:10 smithi106 ceph-mon[25742]: rocksdb: Options.statistics: (nil) 2024-03-10T00:46:10.342 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:46:10 smithi106 ceph-mon[25742]: rocksdb: Options.use_fsync: 0 2024-03-10T00:46:10.342 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:46:10 smithi106 ceph-mon[25742]: rocksdb: Options.max_log_file_size: 0 2024-03-10T00:46:10.342 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:46:10 smithi106 ceph-mon[25742]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-03-10T00:46:10.342 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:46:10 smithi106 ceph-mon[25742]: rocksdb: Options.log_file_time_to_roll: 0 2024-03-10T00:46:10.342 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:46:10 smithi106 ceph-mon[25742]: rocksdb: Options.keep_log_file_num: 1000 2024-03-10T00:46:10.342 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:46:10 smithi106 ceph-mon[25742]: rocksdb: Options.recycle_log_file_num: 0 2024-03-10T00:46:10.342 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:46:10 smithi106 ceph-mon[25742]: rocksdb: Options.allow_fallocate: 1 2024-03-10T00:46:10.342 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:46:10 smithi106 ceph-mon[25742]: rocksdb: Options.allow_mmap_reads: 0 2024-03-10T00:46:10.342 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:46:10 smithi106 ceph-mon[25742]: rocksdb: Options.allow_mmap_writes: 0 2024-03-10T00:46:10.342 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:46:10 smithi106 ceph-mon[25742]: rocksdb: Options.use_direct_reads: 0 2024-03-10T00:46:10.342 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:46:10 smithi106 ceph-mon[25742]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-03-10T00:46:10.342 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:46:10 smithi106 ceph-mon[25742]: rocksdb: Options.create_missing_column_families: 0 2024-03-10T00:46:10.343 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:46:10 smithi106 ceph-mon[25742]: rocksdb: Options.db_log_dir: 2024-03-10T00:46:10.343 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:46:10 smithi106 ceph-mon[25742]: rocksdb: Options.wal_dir: 2024-03-10T00:46:10.343 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:46:10 smithi106 ceph-mon[25742]: rocksdb: Options.table_cache_numshardbits: 6 2024-03-10T00:46:10.343 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:46:10 smithi106 ceph-mon[25742]: rocksdb: Options.WAL_ttl_seconds: 0 2024-03-10T00:46:10.343 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:46:10 smithi106 ceph-mon[25742]: rocksdb: Options.WAL_size_limit_MB: 0 2024-03-10T00:46:10.343 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:46:10 smithi106 ceph-mon[25742]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-03-10T00:46:10.343 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:46:10 smithi106 ceph-mon[25742]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-03-10T00:46:10.343 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:46:10 smithi106 ceph-mon[25742]: rocksdb: Options.is_fd_close_on_exec: 1 2024-03-10T00:46:10.343 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:46:10 smithi106 ceph-mon[25742]: rocksdb: Options.advise_random_on_open: 1 2024-03-10T00:46:10.343 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:46:10 smithi106 ceph-mon[25742]: rocksdb: Options.db_write_buffer_size: 0 2024-03-10T00:46:10.343 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:46:10 smithi106 ceph-mon[25742]: rocksdb: Options.write_buffer_manager: 0x5578570b9400 2024-03-10T00:46:10.343 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:46:10 smithi106 ceph-mon[25742]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-03-10T00:46:10.343 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:46:10 smithi106 ceph-mon[25742]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-03-10T00:46:10.343 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:46:10 smithi106 ceph-mon[25742]: rocksdb: Options.use_adaptive_mutex: 0 2024-03-10T00:46:10.344 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:46:10 smithi106 ceph-mon[25742]: rocksdb: Options.rate_limiter: (nil) 2024-03-10T00:46:10.344 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:46:10 smithi106 ceph-mon[25742]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-03-10T00:46:10.344 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:46:10 smithi106 ceph-mon[25742]: rocksdb: Options.wal_recovery_mode: 2 2024-03-10T00:46:10.344 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:46:10 smithi106 ceph-mon[25742]: rocksdb: Options.enable_thread_tracking: 0 2024-03-10T00:46:10.344 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:46:10 smithi106 ceph-mon[25742]: rocksdb: Options.enable_pipelined_write: 0 2024-03-10T00:46:10.344 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:46:10 smithi106 ceph-mon[25742]: rocksdb: Options.unordered_write: 0 2024-03-10T00:46:10.344 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:46:10 smithi106 ceph-mon[25742]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-03-10T00:46:10.344 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:46:10 smithi106 ceph-mon[25742]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-03-10T00:46:10.344 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:46:10 smithi106 ceph-mon[25742]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-03-10T00:46:10.344 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:46:10 smithi106 ceph-mon[25742]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-03-10T00:46:10.344 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:46:10 smithi106 ceph-mon[25742]: rocksdb: Options.row_cache: None 2024-03-10T00:46:10.344 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:46:10 smithi106 ceph-mon[25742]: rocksdb: Options.wal_filter: None 2024-03-10T00:46:10.346 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:46:10 smithi106 ceph-mon[25742]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-03-10T00:46:10.346 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:46:10 smithi106 ceph-mon[25742]: rocksdb: Options.allow_ingest_behind: 0 2024-03-10T00:46:10.346 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:46:10 smithi106 ceph-mon[25742]: rocksdb: Options.two_write_queues: 0 2024-03-10T00:46:10.346 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:46:10 smithi106 ceph-mon[25742]: rocksdb: Options.manual_wal_flush: 0 2024-03-10T00:46:10.346 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:46:10 smithi106 ceph-mon[25742]: rocksdb: Options.wal_compression: 0 2024-03-10T00:46:10.346 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:46:10 smithi106 ceph-mon[25742]: rocksdb: Options.atomic_flush: 0 2024-03-10T00:46:10.346 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:46:10 smithi106 ceph-mon[25742]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-03-10T00:46:10.346 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:46:10 smithi106 ceph-mon[25742]: rocksdb: Options.persist_stats_to_disk: 0 2024-03-10T00:46:10.346 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:46:10 smithi106 ceph-mon[25742]: rocksdb: Options.write_dbid_to_manifest: 0 2024-03-10T00:46:10.346 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:46:10 smithi106 ceph-mon[25742]: rocksdb: Options.log_readahead_size: 0 2024-03-10T00:46:10.346 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:46:10 smithi106 ceph-mon[25742]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-03-10T00:46:10.346 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:46:10 smithi106 ceph-mon[25742]: rocksdb: Options.best_efforts_recovery: 0 2024-03-10T00:46:10.346 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:46:10 smithi106 ceph-mon[25742]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-03-10T00:46:10.347 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:46:10 smithi106 ceph-mon[25742]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-03-10T00:46:10.347 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:46:10 smithi106 ceph-mon[25742]: rocksdb: Options.allow_data_in_errors: 0 2024-03-10T00:46:10.347 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:46:10 smithi106 ceph-mon[25742]: rocksdb: Options.db_host_id: __hostname__ 2024-03-10T00:46:10.347 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:46:10 smithi106 ceph-mon[25742]: rocksdb: Options.enforce_single_del_contracts: true 2024-03-10T00:46:10.347 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:46:10 smithi106 ceph-mon[25742]: rocksdb: Options.max_background_jobs: 2 2024-03-10T00:46:10.347 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:46:10 smithi106 ceph-mon[25742]: rocksdb: Options.max_background_compactions: -1 2024-03-10T00:46:10.347 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:46:10 smithi106 ceph-mon[25742]: rocksdb: Options.max_subcompactions: 1 2024-03-10T00:46:10.347 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:46:10 smithi106 ceph-mon[25742]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-03-10T00:46:10.347 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:46:10 smithi106 ceph-mon[25742]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-03-10T00:46:10.347 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:46:10 smithi106 ceph-mon[25742]: rocksdb: Options.delayed_write_rate : 16777216 2024-03-10T00:46:10.347 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:46:10 smithi106 ceph-mon[25742]: rocksdb: Options.max_total_wal_size: 0 2024-03-10T00:46:10.347 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:46:10 smithi106 ceph-mon[25742]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-03-10T00:46:10.347 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:46:10 smithi106 ceph-mon[25742]: rocksdb: Options.stats_dump_period_sec: 600 2024-03-10T00:46:10.347 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:46:10 smithi106 ceph-mon[25742]: rocksdb: Options.stats_persist_period_sec: 600 2024-03-10T00:46:10.348 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:46:10 smithi106 ceph-mon[25742]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-03-10T00:46:10.348 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:46:10 smithi106 ceph-mon[25742]: rocksdb: Options.max_open_files: -1 2024-03-10T00:46:10.348 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:46:10 smithi106 ceph-mon[25742]: rocksdb: Options.bytes_per_sync: 0 2024-03-10T00:46:10.348 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:46:10 smithi106 ceph-mon[25742]: rocksdb: Options.wal_bytes_per_sync: 0 2024-03-10T00:46:10.348 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:46:10 smithi106 ceph-mon[25742]: rocksdb: Options.strict_bytes_per_sync: 0 2024-03-10T00:46:10.348 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:46:10 smithi106 ceph-mon[25742]: rocksdb: Options.compaction_readahead_size: 0 2024-03-10T00:46:10.348 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:46:10 smithi106 ceph-mon[25742]: rocksdb: Options.max_background_flushes: -1 2024-03-10T00:46:10.348 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:46:10 smithi106 ceph-mon[25742]: rocksdb: Compression algorithms supported: 2024-03-10T00:46:10.348 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:46:10 smithi106 ceph-mon[25742]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-03-10T00:46:10.348 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:46:10 smithi106 ceph-mon[25742]: rocksdb: kZSTD supported: 0 2024-03-10T00:46:10.348 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:46:10 smithi106 ceph-mon[25742]: rocksdb: kXpressCompression supported: 0 2024-03-10T00:46:10.348 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:46:10 smithi106 ceph-mon[25742]: rocksdb: kLZ4HCCompression supported: 1 2024-03-10T00:46:10.348 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:46:10 smithi106 ceph-mon[25742]: rocksdb: kZlibCompression supported: 1 2024-03-10T00:46:10.348 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:46:10 smithi106 ceph-mon[25742]: rocksdb: kSnappyCompression supported: 1 2024-03-10T00:46:10.349 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:46:10 smithi106 ceph-mon[25742]: rocksdb: kLZ4Compression supported: 1 2024-03-10T00:46:10.349 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:46:10 smithi106 ceph-mon[25742]: rocksdb: kBZip2Compression supported: 0 2024-03-10T00:46:10.349 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:46:10 smithi106 ceph-mon[25742]: rocksdb: Fast CRC32 supported: Supported on x86 2024-03-10T00:46:10.349 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:46:10 smithi106 ceph-mon[25742]: rocksdb: DMutex implementation: pthread_mutex_t 2024-03-10T00:46:10.349 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:46:10 smithi106 ceph-mon[25742]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000010 2024-03-10T00:46:10.349 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:46:10 smithi106 ceph-mon[25742]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-03-10T00:46:10.349 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:46:10 smithi106 ceph-mon[25742]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-03-10T00:46:10.349 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:46:10 smithi106 ceph-mon[25742]: rocksdb: Options.merge_operator: 2024-03-10T00:46:10.349 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:46:10 smithi106 ceph-mon[25742]: rocksdb: Options.compaction_filter: None 2024-03-10T00:46:10.349 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:46:10 smithi106 ceph-mon[25742]: rocksdb: Options.compaction_filter_factory: None 2024-03-10T00:46:10.349 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:46:10 smithi106 ceph-mon[25742]: rocksdb: Options.sst_partitioner_factory: None 2024-03-10T00:46:10.349 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:46:10 smithi106 ceph-mon[25742]: rocksdb: Options.memtable_factory: SkipListFactory 2024-03-10T00:46:10.349 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:46:10 smithi106 ceph-mon[25742]: rocksdb: Options.table_factory: BlockBasedTable 2024-03-10T00:46:10.349 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:46:10 smithi106 ceph-mon[25742]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x5578570e59e0) 2024-03-10T00:46:10.350 INFO:journalctl@ceph.mon.a.smithi106.stdout: cache_index_and_filter_blocks: 1 2024-03-10T00:46:10.350 INFO:journalctl@ceph.mon.a.smithi106.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-03-10T00:46:10.350 INFO:journalctl@ceph.mon.a.smithi106.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-03-10T00:46:10.350 INFO:journalctl@ceph.mon.a.smithi106.stdout: pin_top_level_index_and_filter: 1 2024-03-10T00:46:10.350 INFO:journalctl@ceph.mon.a.smithi106.stdout: index_type: 0 2024-03-10T00:46:10.350 INFO:journalctl@ceph.mon.a.smithi106.stdout: data_block_index_type: 0 2024-03-10T00:46:10.350 INFO:journalctl@ceph.mon.a.smithi106.stdout: index_shortening: 1 2024-03-10T00:46:10.350 INFO:journalctl@ceph.mon.a.smithi106.stdout: data_block_hash_table_util_ratio: 0.750000 2024-03-10T00:46:10.350 INFO:journalctl@ceph.mon.a.smithi106.stdout: checksum: 4 2024-03-10T00:46:10.350 INFO:journalctl@ceph.mon.a.smithi106.stdout: no_block_cache: 0 2024-03-10T00:46:10.350 INFO:journalctl@ceph.mon.a.smithi106.stdout: block_cache: 0x5578570eb090 2024-03-10T00:46:10.350 INFO:journalctl@ceph.mon.a.smithi106.stdout: block_cache_name: BinnedLRUCache 2024-03-10T00:46:10.350 INFO:journalctl@ceph.mon.a.smithi106.stdout: block_cache_options: 2024-03-10T00:46:10.350 INFO:journalctl@ceph.mon.a.smithi106.stdout: capacity : 536870912 2024-03-10T00:46:10.351 INFO:journalctl@ceph.mon.a.smithi106.stdout: num_shard_bits : 4 2024-03-10T00:46:10.351 INFO:journalctl@ceph.mon.a.smithi106.stdout: strict_capacity_limit : 0 2024-03-10T00:46:10.351 INFO:journalctl@ceph.mon.a.smithi106.stdout: high_pri_pool_ratio: 0.000 2024-03-10T00:46:10.351 INFO:journalctl@ceph.mon.a.smithi106.stdout: block_cache_compressed: (nil) 2024-03-10T00:46:10.351 INFO:journalctl@ceph.mon.a.smithi106.stdout: persistent_cache: (nil) 2024-03-10T00:46:10.351 INFO:journalctl@ceph.mon.a.smithi106.stdout: block_size: 4096 2024-03-10T00:46:10.351 INFO:journalctl@ceph.mon.a.smithi106.stdout: block_size_deviation: 10 2024-03-10T00:46:10.351 INFO:journalctl@ceph.mon.a.smithi106.stdout: block_restart_interval: 16 2024-03-10T00:46:10.351 INFO:journalctl@ceph.mon.a.smithi106.stdout: index_block_restart_interval: 1 2024-03-10T00:46:10.351 INFO:journalctl@ceph.mon.a.smithi106.stdout: metadata_block_size: 4096 2024-03-10T00:46:10.351 INFO:journalctl@ceph.mon.a.smithi106.stdout: partition_filters: 0 2024-03-10T00:46:10.351 INFO:journalctl@ceph.mon.a.smithi106.stdout: use_delta_encoding: 1 2024-03-10T00:46:10.351 INFO:journalctl@ceph.mon.a.smithi106.stdout: filter_policy: bloomfilter 2024-03-10T00:46:10.351 INFO:journalctl@ceph.mon.a.smithi106.stdout: whole_key_filtering: 1 2024-03-10T00:46:10.351 INFO:journalctl@ceph.mon.a.smithi106.stdout: verify_compression: 0 2024-03-10T00:46:10.352 INFO:journalctl@ceph.mon.a.smithi106.stdout: read_amp_bytes_per_bit: 0 2024-03-10T00:46:10.352 INFO:journalctl@ceph.mon.a.smithi106.stdout: format_version: 5 2024-03-10T00:46:10.352 INFO:journalctl@ceph.mon.a.smithi106.stdout: enable_index_compression: 1 2024-03-10T00:46:10.352 INFO:journalctl@ceph.mon.a.smithi106.stdout: block_align: 0 2024-03-10T00:46:10.352 INFO:journalctl@ceph.mon.a.smithi106.stdout: max_auto_readahead_size: 262144 2024-03-10T00:46:10.352 INFO:journalctl@ceph.mon.a.smithi106.stdout: prepopulate_block_cache: 0 2024-03-10T00:46:10.352 INFO:journalctl@ceph.mon.a.smithi106.stdout: initial_auto_readahead_size: 8192 2024-03-10T00:46:10.352 INFO:journalctl@ceph.mon.a.smithi106.stdout: num_file_reads_for_auto_readahead: 2 2024-03-10T00:46:10.352 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:46:10 smithi106 ceph-mon[25742]: rocksdb: Options.write_buffer_size: 33554432 2024-03-10T00:46:10.352 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:46:10 smithi106 ceph-mon[25742]: rocksdb: Options.max_write_buffer_number: 2 2024-03-10T00:46:10.352 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:46:10 smithi106 ceph-mon[25742]: rocksdb: Options.compression: NoCompression 2024-03-10T00:46:10.352 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:46:10 smithi106 ceph-mon[25742]: rocksdb: Options.bottommost_compression: Disabled 2024-03-10T00:46:10.352 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:46:10 smithi106 ceph-mon[25742]: rocksdb: Options.prefix_extractor: nullptr 2024-03-10T00:46:10.352 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:46:10 smithi106 ceph-mon[25742]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-03-10T00:46:10.353 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:46:10 smithi106 ceph-mon[25742]: rocksdb: Options.num_levels: 7 2024-03-10T00:46:10.353 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:46:10 smithi106 ceph-mon[25742]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-03-10T00:46:10.353 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:46:10 smithi106 ceph-mon[25742]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-03-10T00:46:10.353 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:46:10 smithi106 ceph-mon[25742]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-03-10T00:46:10.353 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:46:10 smithi106 ceph-mon[25742]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-03-10T00:46:10.353 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:46:10 smithi106 ceph-mon[25742]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-03-10T00:46:10.353 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:46:10 smithi106 ceph-mon[25742]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-03-10T00:46:10.353 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:46:10 smithi106 ceph-mon[25742]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-03-10T00:46:10.353 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:46:10 smithi106 ceph-mon[25742]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-03-10T00:46:10.353 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:46:10 smithi106 ceph-mon[25742]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-03-10T00:46:10.353 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:46:10 smithi106 ceph-mon[25742]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-03-10T00:46:10.353 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:46:10 smithi106 ceph-mon[25742]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-03-10T00:46:10.353 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:46:10 smithi106 ceph-mon[25742]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-03-10T00:46:10.353 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:46:10 smithi106 ceph-mon[25742]: rocksdb: Options.compression_opts.window_bits: -14 2024-03-10T00:46:10.354 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:46:10 smithi106 ceph-mon[25742]: rocksdb: Options.compression_opts.level: 32767 2024-03-10T00:46:10.354 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:46:10 smithi106 ceph-mon[25742]: rocksdb: Options.compression_opts.strategy: 0 2024-03-10T00:46:10.354 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:46:10 smithi106 ceph-mon[25742]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-03-10T00:46:10.354 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:46:10 smithi106 ceph-mon[25742]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-03-10T00:46:10.354 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:46:10 smithi106 ceph-mon[25742]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-03-10T00:46:10.354 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:46:10 smithi106 ceph-mon[25742]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-03-10T00:46:10.354 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:46:10 smithi106 ceph-mon[25742]: rocksdb: Options.compression_opts.enabled: false 2024-03-10T00:46:10.354 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:46:10 smithi106 ceph-mon[25742]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-03-10T00:46:10.354 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:46:10 smithi106 ceph-mon[25742]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-03-10T00:46:10.354 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:46:10 smithi106 ceph-mon[25742]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-03-10T00:46:10.354 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:46:10 smithi106 ceph-mon[25742]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-03-10T00:46:10.354 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:46:10 smithi106 ceph-mon[25742]: rocksdb: Options.target_file_size_base: 67108864 2024-03-10T00:46:10.354 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:46:10 smithi106 ceph-mon[25742]: rocksdb: Options.target_file_size_multiplier: 1 2024-03-10T00:46:10.354 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:46:10 smithi106 ceph-mon[25742]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-03-10T00:46:10.355 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:46:10 smithi106 ceph-mon[25742]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-03-10T00:46:10.355 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:46:10 smithi106 ceph-mon[25742]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-03-10T00:46:10.355 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:46:10 smithi106 ceph-mon[25742]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-03-10T00:46:10.355 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:46:10 smithi106 ceph-mon[25742]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-03-10T00:46:10.355 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:46:10 smithi106 ceph-mon[25742]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-03-10T00:46:10.355 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:46:10 smithi106 ceph-mon[25742]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-03-10T00:46:10.355 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:46:10 smithi106 ceph-mon[25742]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-03-10T00:46:10.355 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:46:10 smithi106 ceph-mon[25742]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-03-10T00:46:10.355 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:46:10 smithi106 ceph-mon[25742]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-03-10T00:46:10.355 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:46:10 smithi106 ceph-mon[25742]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-03-10T00:46:10.355 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:46:10 smithi106 ceph-mon[25742]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-03-10T00:46:10.355 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:46:10 smithi106 ceph-mon[25742]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-03-10T00:46:10.355 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:46:10 smithi106 ceph-mon[25742]: rocksdb: Options.arena_block_size: 1048576 2024-03-10T00:46:10.356 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:46:10 smithi106 ceph-mon[25742]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-03-10T00:46:10.356 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:46:10 smithi106 ceph-mon[25742]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-03-10T00:46:10.356 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:46:10 smithi106 ceph-mon[25742]: rocksdb: Options.disable_auto_compactions: 0 2024-03-10T00:46:10.356 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:46:10 smithi106 ceph-mon[25742]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-03-10T00:46:10.356 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:46:10 smithi106 ceph-mon[25742]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-03-10T00:46:10.356 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:46:10 smithi106 ceph-mon[25742]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-03-10T00:46:10.356 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:46:10 smithi106 ceph-mon[25742]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-03-10T00:46:10.356 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:46:10 smithi106 ceph-mon[25742]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-03-10T00:46:10.356 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:46:10 smithi106 ceph-mon[25742]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-03-10T00:46:10.356 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:46:10 smithi106 ceph-mon[25742]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-03-10T00:46:10.356 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:46:10 smithi106 ceph-mon[25742]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-03-10T00:46:10.356 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:46:10 smithi106 ceph-mon[25742]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-03-10T00:46:10.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:46:10 smithi106 ceph-mon[25742]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-03-10T00:46:10.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:46:10 smithi106 ceph-mon[25742]: rocksdb: Options.table_properties_collectors: 2024-03-10T00:46:10.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:46:10 smithi106 ceph-mon[25742]: rocksdb: Options.inplace_update_support: 0 2024-03-10T00:46:10.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:46:10 smithi106 ceph-mon[25742]: rocksdb: Options.inplace_update_num_locks: 10000 2024-03-10T00:46:10.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:46:10 smithi106 ceph-mon[25742]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-03-10T00:46:10.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:46:10 smithi106 ceph-mon[25742]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-03-10T00:46:10.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:46:10 smithi106 ceph-mon[25742]: rocksdb: Options.memtable_huge_page_size: 0 2024-03-10T00:46:10.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:46:10 smithi106 ceph-mon[25742]: rocksdb: Options.bloom_locality: 0 2024-03-10T00:46:10.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:46:10 smithi106 ceph-mon[25742]: rocksdb: Options.max_successive_merges: 0 2024-03-10T00:46:10.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:46:10 smithi106 ceph-mon[25742]: rocksdb: Options.optimize_filters_for_hits: 0 2024-03-10T00:46:10.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:46:10 smithi106 ceph-mon[25742]: rocksdb: Options.paranoid_file_checks: 0 2024-03-10T00:46:10.358 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:46:10 smithi106 ceph-mon[25742]: rocksdb: Options.force_consistency_checks: 1 2024-03-10T00:46:10.358 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:46:10 smithi106 ceph-mon[25742]: rocksdb: Options.report_bg_io_stats: 0 2024-03-10T00:46:10.358 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:46:10 smithi106 ceph-mon[25742]: rocksdb: Options.ttl: 2592000 2024-03-10T00:46:10.358 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:46:10 smithi106 ceph-mon[25742]: rocksdb: Options.periodic_compaction_seconds: 0 2024-03-10T00:46:10.358 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:46:10 smithi106 ceph-mon[25742]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-03-10T00:46:10.358 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:46:10 smithi106 ceph-mon[25742]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-03-10T00:46:10.358 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:46:10 smithi106 ceph-mon[25742]: rocksdb: Options.enable_blob_files: false 2024-03-10T00:46:10.358 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:46:10 smithi106 ceph-mon[25742]: rocksdb: Options.min_blob_size: 0 2024-03-10T00:46:10.359 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:46:10 smithi106 ceph-mon[25742]: rocksdb: Options.blob_file_size: 268435456 2024-03-10T00:46:10.359 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:46:10 smithi106 ceph-mon[25742]: rocksdb: Options.blob_compression_type: NoCompression 2024-03-10T00:46:10.359 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:46:10 smithi106 ceph-mon[25742]: rocksdb: Options.enable_blob_garbage_collection: false 2024-03-10T00:46:10.359 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:46:10 smithi106 ceph-mon[25742]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-03-10T00:46:10.359 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:46:10 smithi106 ceph-mon[25742]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-03-10T00:46:10.359 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:46:10 smithi106 ceph-mon[25742]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-03-10T00:46:10.359 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:46:10 smithi106 ceph-mon[25742]: rocksdb: Options.blob_file_starting_level: 0 2024-03-10T00:46:10.359 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:46:10 smithi106 ceph-mon[25742]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-03-10T00:46:10.359 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:46:10 smithi106 ceph-mon[25742]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000010 succeeded,manifest_file_number is 10, next_file_number is 12, last_sequence is 5, log_number is 5,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 5 2024-03-10T00:46:10.360 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:46:10 smithi106 ceph-mon[25742]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 5 2024-03-10T00:46:10.360 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:46:10 smithi106 ceph-mon[25742]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: e514fcbf-e6f9-4ca7-b286-71fb62ce27f8 2024-03-10T00:46:10.360 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:46:10 smithi106 ceph-mon[25742]: rocksdb: EVENT_LOG_v1 {"time_micros": 1710031570124528, "job": 1, "event": "recovery_started", "wal_files": [9]} 2024-03-10T00:46:10.360 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:46:10 smithi106 ceph-mon[25742]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #9 mode 2 2024-03-10T00:46:10.360 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:46:10 smithi106 ceph-mon[25742]: rocksdb: EVENT_LOG_v1 {"time_micros": 1710031570125445, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 13, "file_size": 81466, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 8, "largest_seqno": 258, "table_properties": {"data_size": 79635, "index_size": 241, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 581, "raw_key_size": 10057, "raw_average_key_size": 47, "raw_value_size": 73899, "raw_average_value_size": 346, "num_data_blocks": 11, "num_entries": 213, "num_filter_entries": 213, "num_deletions": 3, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1710031570, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "e514fcbf-e6f9-4ca7-b286-71fb62ce27f8", "db_session_id": "CTU7EGIYN8XQK7H0GSN2", "orig_file_number": 13, "seqno_to_time_mapping": "N/A"}} 2024-03-10T00:46:10.360 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:46:10 smithi106 ceph-mon[25742]: rocksdb: EVENT_LOG_v1 {"time_micros": 1710031570125510, "job": 1, "event": "recovery_finished"} 2024-03-10T00:46:10.360 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:46:10 smithi106 ceph-mon[25742]: rocksdb: [db/version_set.cc:5047] Creating manifest 15 2024-03-10T00:46:10.360 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:46:10 smithi106 ceph-mon[25742]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-a/store.db/000009.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-03-10T00:46:10.361 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:46:10 smithi106 ceph-mon[25742]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x5578571de000 2024-03-10T00:46:10.361 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:46:10 smithi106 ceph-mon[25742]: rocksdb: DB pointer 0x5578571c8000 2024-03-10T00:46:10.361 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:46:10 smithi106 ceph-mon[25742]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-03-10T00:46:10.361 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:46:10 smithi106 ceph-mon[25742]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-03-10T00:46:10.361 INFO:journalctl@ceph.mon.a.smithi106.stdout: ** DB Stats ** 2024-03-10T00:46:10.361 INFO:journalctl@ceph.mon.a.smithi106.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-03-10T00:46:10.361 INFO:journalctl@ceph.mon.a.smithi106.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-03-10T00:46:10.361 INFO:journalctl@ceph.mon.a.smithi106.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-03-10T00:46:10.361 INFO:journalctl@ceph.mon.a.smithi106.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-03-10T00:46:10.362 INFO:journalctl@ceph.mon.a.smithi106.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-03-10T00:46:10.362 INFO:journalctl@ceph.mon.a.smithi106.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-03-10T00:46:10.362 INFO:journalctl@ceph.mon.a.smithi106.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-03-10T00:46:10.362 INFO:journalctl@ceph.mon.a.smithi106.stdout: 2024-03-10T00:46:10.362 INFO:journalctl@ceph.mon.a.smithi106.stdout: ** Compaction Stats [default] ** 2024-03-10T00:46:10.362 INFO:journalctl@ceph.mon.a.smithi106.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-03-10T00:46:10.362 INFO:journalctl@ceph.mon.a.smithi106.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-03-10T00:46:10.363 INFO:journalctl@ceph.mon.a.smithi106.stdout: L0 2/0 81.39 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 134.2 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-03-10T00:46:10.363 INFO:journalctl@ceph.mon.a.smithi106.stdout: Sum 2/0 81.39 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 134.2 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-03-10T00:46:10.363 INFO:journalctl@ceph.mon.a.smithi106.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 134.2 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-03-10T00:46:10.363 INFO:journalctl@ceph.mon.a.smithi106.stdout: 2024-03-10T00:46:10.363 INFO:journalctl@ceph.mon.a.smithi106.stdout: ** Compaction Stats [default] ** 2024-03-10T00:46:10.363 INFO:journalctl@ceph.mon.a.smithi106.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-03-10T00:46:10.363 INFO:journalctl@ceph.mon.a.smithi106.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-03-10T00:46:10.363 INFO:journalctl@ceph.mon.a.smithi106.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 134.2 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-03-10T00:46:10.363 INFO:journalctl@ceph.mon.a.smithi106.stdout: 2024-03-10T00:46:10.363 INFO:journalctl@ceph.mon.a.smithi106.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-03-10T00:46:10.363 INFO:journalctl@ceph.mon.a.smithi106.stdout: 2024-03-10T00:46:10.363 INFO:journalctl@ceph.mon.a.smithi106.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-03-10T00:46:10.363 INFO:journalctl@ceph.mon.a.smithi106.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-03-10T00:46:10.364 INFO:journalctl@ceph.mon.a.smithi106.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-03-10T00:46:10.364 INFO:journalctl@ceph.mon.a.smithi106.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-03-10T00:46:10.364 INFO:journalctl@ceph.mon.a.smithi106.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-03-10T00:46:10.364 INFO:journalctl@ceph.mon.a.smithi106.stdout: AddFile(Keys): cumulative 0, interval 0 2024-03-10T00:46:10.364 INFO:journalctl@ceph.mon.a.smithi106.stdout: Cumulative compaction: 0.00 GB write, 25.11 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-10T00:46:10.364 INFO:journalctl@ceph.mon.a.smithi106.stdout: Interval compaction: 0.00 GB write, 25.11 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-10T00:46:10.364 INFO:journalctl@ceph.mon.a.smithi106.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-03-10T00:46:10.364 INFO:journalctl@ceph.mon.a.smithi106.stdout: Block cache BinnedLRUCache@0x5578570eb090#7 capacity: 512.00 MB usage: 1.20 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 1e-05 secs_since: 0 2024-03-10T00:46:10.364 INFO:journalctl@ceph.mon.a.smithi106.stdout: Block cache entry stats(count,size,portion): FilterBlock(2,0.77 KB,0.000146031%) IndexBlock(2,0.44 KB,8.34465e-05%) Misc(1,0.00 KB,0%) 2024-03-10T00:46:10.364 INFO:journalctl@ceph.mon.a.smithi106.stdout: 2024-03-10T00:46:10.364 INFO:journalctl@ceph.mon.a.smithi106.stdout: ** File Read Latency Histogram By Level [default] ** 2024-03-10T00:46:10.364 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:46:10 smithi106 ceph-mon[25742]: starting mon.a rank 0 at public addrs [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] at bind addrs [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] mon_data /var/lib/ceph/mon/ceph-a fsid 75c37498-de77-11ee-95c7-87774f69a715 2024-03-10T00:46:10.364 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:46:10 smithi106 ceph-mon[25742]: mon.a@-1(???) e1 preinit fsid 75c37498-de77-11ee-95c7-87774f69a715 2024-03-10T00:46:10.364 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:46:10 smithi106 ceph-mon[25742]: mon.a@-1(???).mds e1 new map 2024-03-10T00:46:10.365 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:46:10 smithi106 ceph-mon[25742]: mon.a@-1(???).mds e1 print_map 2024-03-10T00:46:10.365 INFO:journalctl@ceph.mon.a.smithi106.stdout: e1 2024-03-10T00:46:10.365 INFO:journalctl@ceph.mon.a.smithi106.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-03-10T00:46:10.365 INFO:journalctl@ceph.mon.a.smithi106.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments} 2024-03-10T00:46:10.365 INFO:journalctl@ceph.mon.a.smithi106.stdout: legacy client fscid: -1 2024-03-10T00:46:10.365 INFO:journalctl@ceph.mon.a.smithi106.stdout: 2024-03-10T00:46:10.365 INFO:journalctl@ceph.mon.a.smithi106.stdout: No filesystems configured 2024-03-10T00:46:10.365 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:46:10 smithi106 ceph-mon[25742]: mon.a@-1(???).osd e1 crush map has features 3314932999778484224, adjusting msgr requires 2024-03-10T00:46:10.365 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:46:10 smithi106 ceph-mon[25742]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-03-10T00:46:10.365 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:46:10 smithi106 ceph-mon[25742]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-03-10T00:46:10.365 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:46:10 smithi106 ceph-mon[25742]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-03-10T00:46:10.365 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:46:10 smithi106 ceph-mon[25742]: mon.a@-1(???).paxosservice(auth 1..2) refresh upgraded, format 0 -> 3 2024-03-10T00:46:10.365 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:46:10 smithi106 ceph-mon[25742]: mon.a is new leader, mons a in quorum (ranks 0) 2024-03-10T00:46:10.365 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:46:10 smithi106 ceph-mon[25742]: monmap e1: 1 mons at {a=[v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-10T00:46:10.366 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:46:10 smithi106 ceph-mon[25742]: fsmap 2024-03-10T00:46:10.366 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:46:10 smithi106 ceph-mon[25742]: osdmap e1: 0 total, 0 up, 0 in 2024-03-10T00:46:10.366 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:46:10 smithi106 ceph-mon[25742]: mgrmap e1: no daemons active 2024-03-10T00:46:10.954 INFO:teuthology.orchestra.run.smithi106.stdout:Setting public_network to 172.21.0.0/20 in mon config section 2024-03-10T00:46:11.339 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:46:10 smithi106 bash[25704]: 97a03d7ed61790350cb5cb41bd58fe8d1f02ce8a94e85b2c0994a6ae2a89679b 2024-03-10T00:46:11.339 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:46:10 smithi106 systemd[1]: Started Ceph mon.a for 75c37498-de77-11ee-95c7-87774f69a715. 2024-03-10T00:46:13.086 INFO:teuthology.orchestra.run.smithi106.stdout:Wrote config to /etc/ceph/ceph.conf 2024-03-10T00:46:13.100 INFO:teuthology.orchestra.run.smithi106.stdout:Wrote keyring to /etc/ceph/ceph.client.admin.keyring 2024-03-10T00:46:13.100 INFO:teuthology.orchestra.run.smithi106.stdout:Creating mgr... 2024-03-10T00:46:13.100 INFO:teuthology.orchestra.run.smithi106.stdout:Verifying port 0.0.0.0:9283 ... 2024-03-10T00:46:13.101 INFO:teuthology.orchestra.run.smithi106.stdout:Verifying port 0.0.0.0:8765 ... 2024-03-10T00:46:13.340 INFO:teuthology.orchestra.run.smithi106.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-75c37498-de77-11ee-95c7-87774f69a715@mgr.y 2024-03-10T00:46:13.340 INFO:teuthology.orchestra.run.smithi106.stdout:systemctl: stderr Failed to reset failed state of unit ceph-75c37498-de77-11ee-95c7-87774f69a715@mgr.y.service: Unit ceph-75c37498-de77-11ee-95c7-87774f69a715@mgr.y.service not loaded. 2024-03-10T00:46:13.491 INFO:teuthology.orchestra.run.smithi106.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-75c37498-de77-11ee-95c7-87774f69a715.target.wants/ceph-75c37498-de77-11ee-95c7-87774f69a715@mgr.y.service → /etc/systemd/system/ceph-75c37498-de77-11ee-95c7-87774f69a715@.service. 2024-03-10T00:46:13.839 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:46:13 smithi106 ceph-mon[25742]: from='client.? 172.21.15.106:0/1782217849' entity='client.admin' 2024-03-10T00:46:13.839 INFO:journalctl@ceph.mgr.y.smithi106.stdout:Mar 10 00:46:13 smithi106 podman[25982]: 2024-03-10 00:46:13.62990774 +0000 UTC m=+0.017479476 image pull 01a4f6a6bb3e92ab9ab39a2f835684ea9b10e6ec9fb81ac08ebea4165583a538 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:831bc188fa69f143bbba69ac76119e784500e8e3 2024-03-10T00:46:14.070 INFO:teuthology.orchestra.run.smithi106.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-03-10T00:46:14.070 INFO:teuthology.orchestra.run.smithi106.stdout:systemctl: stdout disabled 2024-03-10T00:46:14.078 INFO:teuthology.orchestra.run.smithi106.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-03-10T00:46:14.078 INFO:teuthology.orchestra.run.smithi106.stdout:systemctl: stdout inactive 2024-03-10T00:46:14.079 INFO:teuthology.orchestra.run.smithi106.stdout:firewalld.service is not enabled 2024-03-10T00:46:14.079 INFO:teuthology.orchestra.run.smithi106.stdout:Not possible to enable service . firewalld.service is not available 2024-03-10T00:46:14.088 INFO:teuthology.orchestra.run.smithi106.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-03-10T00:46:14.089 INFO:teuthology.orchestra.run.smithi106.stdout:systemctl: stdout disabled 2024-03-10T00:46:14.097 INFO:teuthology.orchestra.run.smithi106.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-03-10T00:46:14.097 INFO:teuthology.orchestra.run.smithi106.stdout:systemctl: stdout inactive 2024-03-10T00:46:14.097 INFO:teuthology.orchestra.run.smithi106.stdout:firewalld.service is not enabled 2024-03-10T00:46:14.097 INFO:teuthology.orchestra.run.smithi106.stdout:Not possible to open ports <[9283, 8765]>. firewalld.service is not available 2024-03-10T00:46:14.097 INFO:teuthology.orchestra.run.smithi106.stdout:Waiting for mgr to start... 2024-03-10T00:46:14.097 INFO:teuthology.orchestra.run.smithi106.stdout:Waiting for mgr... 2024-03-10T00:46:14.221 INFO:journalctl@ceph.mgr.y.smithi106.stdout:Mar 10 00:46:13 smithi106 podman[25982]: 2024-03-10 00:46:13.948654568 +0000 UTC m=+0.336226306 container init 52b050b0e011b0de5eaf2a09f03bb853179ca77c4652d1e8ff867cc0e4013ab7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:831bc188fa69f143bbba69ac76119e784500e8e3, name=ceph-75c37498-de77-11ee-95c7-87774f69a715-mgr-y, RELEASE=wip-yuri4-testing-2024-03-05-0854-831bc18, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240222, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, ceph=True, CEPH_POINT_RELEASE=) 2024-03-10T00:46:14.222 INFO:journalctl@ceph.mgr.y.smithi106.stdout:Mar 10 00:46:13 smithi106 podman[25982]: 2024-03-10 00:46:13.954248314 +0000 UTC m=+0.341820046 container start 52b050b0e011b0de5eaf2a09f03bb853179ca77c4652d1e8ff867cc0e4013ab7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:831bc188fa69f143bbba69ac76119e784500e8e3, name=ceph-75c37498-de77-11ee-95c7-87774f69a715-mgr-y, GIT_BRANCH=HEAD, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, org.label-schema.build-date=20240222, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_CLEAN=True, RELEASE=wip-yuri4-testing-2024-03-05-0854-831bc18, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=) 2024-03-10T00:46:14.222 INFO:journalctl@ceph.mgr.y.smithi106.stdout:Mar 10 00:46:14 smithi106 bash[25982]: 52b050b0e011b0de5eaf2a09f03bb853179ca77c4652d1e8ff867cc0e4013ab7 2024-03-10T00:46:14.222 INFO:journalctl@ceph.mgr.y.smithi106.stdout:Mar 10 00:46:14 smithi106 systemd[1]: Started Ceph mgr.y for 75c37498-de77-11ee-95c7-87774f69a715. 2024-03-10T00:46:14.798 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:46:14 smithi106 ceph-mon[25742]: from='client.? 172.21.15.106:0/1763561177' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-03-10T00:46:14.799 INFO:journalctl@ceph.mgr.y.smithi106.stdout:Mar 10 00:46:14 smithi106 ceph-75c37498-de77-11ee-95c7-87774f69a715-mgr-y[25997]: 2024-03-10T00:46:14.525+0000 7f1904d07200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-10T00:46:14.799 INFO:journalctl@ceph.mgr.y.smithi106.stdout:Mar 10 00:46:14 smithi106 ceph-75c37498-de77-11ee-95c7-87774f69a715-mgr-y[25997]: 2024-03-10T00:46:14.623+0000 7f1904d07200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-10T00:46:15.176 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout 2024-03-10T00:46:15.177 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout { 2024-03-10T00:46:15.177 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout "fsid": "75c37498-de77-11ee-95c7-87774f69a715", 2024-03-10T00:46:15.177 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout "health": { 2024-03-10T00:46:15.177 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-03-10T00:46:15.177 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-03-10T00:46:15.177 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-03-10T00:46:15.177 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout }, 2024-03-10T00:46:15.177 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-03-10T00:46:15.177 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-03-10T00:46:15.177 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout 0 2024-03-10T00:46:15.177 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout ], 2024-03-10T00:46:15.177 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-03-10T00:46:15.177 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout "a" 2024-03-10T00:46:15.178 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout ], 2024-03-10T00:46:15.178 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout "quorum_age": 4, 2024-03-10T00:46:15.178 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout "monmap": { 2024-03-10T00:46:15.178 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-10T00:46:15.178 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-03-10T00:46:15.178 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-03-10T00:46:15.178 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout }, 2024-03-10T00:46:15.178 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-03-10T00:46:15.178 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-10T00:46:15.178 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-03-10T00:46:15.179 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-03-10T00:46:15.179 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-03-10T00:46:15.179 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-03-10T00:46:15.179 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-03-10T00:46:15.179 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-03-10T00:46:15.179 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout }, 2024-03-10T00:46:15.179 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-03-10T00:46:15.179 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-03-10T00:46:15.179 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-03-10T00:46:15.179 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-03-10T00:46:15.179 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-03-10T00:46:15.180 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-03-10T00:46:15.180 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-03-10T00:46:15.180 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-03-10T00:46:15.180 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-03-10T00:46:15.180 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout }, 2024-03-10T00:46:15.180 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-03-10T00:46:15.180 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-10T00:46:15.180 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-03-10T00:46:15.180 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-03-10T00:46:15.180 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout }, 2024-03-10T00:46:15.180 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-03-10T00:46:15.180 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout "available": false, 2024-03-10T00:46:15.180 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-03-10T00:46:15.181 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout "modules": [ 2024-03-10T00:46:15.181 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout "iostat", 2024-03-10T00:46:15.181 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout "nfs", 2024-03-10T00:46:15.181 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout "restful" 2024-03-10T00:46:15.181 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout ], 2024-03-10T00:46:15.181 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-10T00:46:15.181 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout }, 2024-03-10T00:46:15.181 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-03-10T00:46:15.181 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-10T00:46:15.181 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout "modified": "2024-03-10T00:46:03.418794+0000", 2024-03-10T00:46:15.181 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-10T00:46:15.181 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout }, 2024-03-10T00:46:15.182 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-03-10T00:46:15.182 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout } 2024-03-10T00:46:15.182 INFO:teuthology.orchestra.run.smithi106.stdout:mgr not available, waiting (1/15)... 2024-03-10T00:46:15.518 INFO:journalctl@ceph.mgr.y.smithi106.stdout:Mar 10 00:46:15 smithi106 ceph-75c37498-de77-11ee-95c7-87774f69a715-mgr-y[25997]: 2024-03-10T00:46:15.281+0000 7f1904d07200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-10T00:46:15.519 INFO:journalctl@ceph.mgr.y.smithi106.stdout:Mar 10 00:46:15 smithi106 ceph-75c37498-de77-11ee-95c7-87774f69a715-mgr-y[25997]: 2024-03-10T00:46:15.377+0000 7f1904d07200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-10T00:46:15.519 INFO:journalctl@ceph.mgr.y.smithi106.stdout:Mar 10 00:46:15 smithi106 ceph-75c37498-de77-11ee-95c7-87774f69a715-mgr-y[25997]: 2024-03-10T00:46:15.519+0000 7f1904d07200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-10T00:46:15.839 INFO:journalctl@ceph.mgr.y.smithi106.stdout:Mar 10 00:46:15 smithi106 ceph-75c37498-de77-11ee-95c7-87774f69a715-mgr-y[25997]: 2024-03-10T00:46:15.615+0000 7f1904d07200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-10T00:46:15.839 INFO:journalctl@ceph.mgr.y.smithi106.stdout:Mar 10 00:46:15 smithi106 ceph-75c37498-de77-11ee-95c7-87774f69a715-mgr-y[25997]: 2024-03-10T00:46:15.696+0000 7f1904d07200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-10T00:46:16.242 INFO:journalctl@ceph.mgr.y.smithi106.stdout:Mar 10 00:46:15 smithi106 ceph-75c37498-de77-11ee-95c7-87774f69a715-mgr-y[25997]: 2024-03-10T00:46:15.956+0000 7f1904d07200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-10T00:46:16.242 INFO:journalctl@ceph.mgr.y.smithi106.stdout:Mar 10 00:46:16 smithi106 ceph-75c37498-de77-11ee-95c7-87774f69a715-mgr-y[25997]: 2024-03-10T00:46:16.064+0000 7f1904d07200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-10T00:46:16.242 INFO:journalctl@ceph.mgr.y.smithi106.stdout:Mar 10 00:46:16 smithi106 ceph-75c37498-de77-11ee-95c7-87774f69a715-mgr-y[25997]: 2024-03-10T00:46:16.150+0000 7f1904d07200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-10T00:46:16.242 INFO:journalctl@ceph.mgr.y.smithi106.stdout:Mar 10 00:46:16 smithi106 ceph-75c37498-de77-11ee-95c7-87774f69a715-mgr-y[25997]: 2024-03-10T00:46:16.243+0000 7f1904d07200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-10T00:46:16.589 INFO:journalctl@ceph.mgr.y.smithi106.stdout:Mar 10 00:46:16 smithi106 ceph-75c37498-de77-11ee-95c7-87774f69a715-mgr-y[25997]: 2024-03-10T00:46:16.417+0000 7f1904d07200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-10T00:46:17.539 INFO:journalctl@ceph.mgr.y.smithi106.stdout:Mar 10 00:46:17 smithi106 ceph-75c37498-de77-11ee-95c7-87774f69a715-mgr-y[25997]: 2024-03-10T00:46:17.401+0000 7f1904d07200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-10T00:46:17.839 INFO:journalctl@ceph.mgr.y.smithi106.stdout:Mar 10 00:46:17 smithi106 ceph-75c37498-de77-11ee-95c7-87774f69a715-mgr-y[25997]: 2024-03-10T00:46:17.540+0000 7f1904d07200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-10T00:46:18.156 INFO:journalctl@ceph.mgr.y.smithi106.stdout:Mar 10 00:46:17 smithi106 ceph-75c37498-de77-11ee-95c7-87774f69a715-mgr-y[25997]: 2024-03-10T00:46:17.889+0000 7f1904d07200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-10T00:46:18.156 INFO:journalctl@ceph.mgr.y.smithi106.stdout:Mar 10 00:46:17 smithi106 ceph-75c37498-de77-11ee-95c7-87774f69a715-mgr-y[25997]: 2024-03-10T00:46:17.982+0000 7f1904d07200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-10T00:46:18.431 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:46:18 smithi106 ceph-mon[25742]: from='client.? 172.21.15.106:0/391795625' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-03-10T00:46:18.431 INFO:journalctl@ceph.mgr.y.smithi106.stdout:Mar 10 00:46:18 smithi106 ceph-75c37498-de77-11ee-95c7-87774f69a715-mgr-y[25997]: 2024-03-10T00:46:18.202+0000 7f1904d07200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-10T00:46:18.431 INFO:journalctl@ceph.mgr.y.smithi106.stdout:Mar 10 00:46:18 smithi106 ceph-75c37498-de77-11ee-95c7-87774f69a715-mgr-y[25997]: 2024-03-10T00:46:18.392+0000 7f1904d07200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-10T00:46:18.783 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout 2024-03-10T00:46:18.783 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout { 2024-03-10T00:46:18.784 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout "fsid": "75c37498-de77-11ee-95c7-87774f69a715", 2024-03-10T00:46:18.784 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout "health": { 2024-03-10T00:46:18.784 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-03-10T00:46:18.784 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-03-10T00:46:18.784 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-03-10T00:46:18.784 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout }, 2024-03-10T00:46:18.784 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-03-10T00:46:18.784 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-03-10T00:46:18.784 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout 0 2024-03-10T00:46:18.784 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout ], 2024-03-10T00:46:18.784 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-03-10T00:46:18.784 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout "a" 2024-03-10T00:46:18.784 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout ], 2024-03-10T00:46:18.784 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout "quorum_age": 8, 2024-03-10T00:46:18.785 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout "monmap": { 2024-03-10T00:46:18.785 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-10T00:46:18.785 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-03-10T00:46:18.785 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-03-10T00:46:18.785 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout }, 2024-03-10T00:46:18.785 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-03-10T00:46:18.785 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-10T00:46:18.785 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-03-10T00:46:18.785 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-03-10T00:46:18.785 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-03-10T00:46:18.785 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-03-10T00:46:18.785 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-03-10T00:46:18.785 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-03-10T00:46:18.785 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout }, 2024-03-10T00:46:18.785 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-03-10T00:46:18.786 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-03-10T00:46:18.786 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-03-10T00:46:18.786 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-03-10T00:46:18.786 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-03-10T00:46:18.786 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-03-10T00:46:18.786 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-03-10T00:46:18.786 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-03-10T00:46:18.786 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-03-10T00:46:18.786 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout }, 2024-03-10T00:46:18.786 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-03-10T00:46:18.786 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-10T00:46:18.786 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-03-10T00:46:18.786 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-03-10T00:46:18.786 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout }, 2024-03-10T00:46:18.787 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-03-10T00:46:18.787 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout "available": false, 2024-03-10T00:46:18.787 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-03-10T00:46:18.787 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout "modules": [ 2024-03-10T00:46:18.787 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout "iostat", 2024-03-10T00:46:18.787 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout "nfs", 2024-03-10T00:46:18.787 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout "restful" 2024-03-10T00:46:18.787 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout ], 2024-03-10T00:46:18.787 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-10T00:46:18.787 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout }, 2024-03-10T00:46:18.787 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-03-10T00:46:18.787 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-10T00:46:18.787 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout "modified": "2024-03-10T00:46:03.418794+0000", 2024-03-10T00:46:18.787 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-10T00:46:18.787 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout }, 2024-03-10T00:46:18.787 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-03-10T00:46:18.788 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout } 2024-03-10T00:46:18.788 INFO:teuthology.orchestra.run.smithi106.stdout:mgr not available, waiting (2/15)... 2024-03-10T00:46:18.943 INFO:journalctl@ceph.mgr.y.smithi106.stdout:Mar 10 00:46:18 smithi106 ceph-75c37498-de77-11ee-95c7-87774f69a715-mgr-y[25997]: 2024-03-10T00:46:18.683+0000 7f1904d07200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-10T00:46:18.943 INFO:journalctl@ceph.mgr.y.smithi106.stdout:Mar 10 00:46:18 smithi106 ceph-75c37498-de77-11ee-95c7-87774f69a715-mgr-y[25997]: 2024-03-10T00:46:18.944+0000 7f1904d07200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-10T00:46:19.339 INFO:journalctl@ceph.mgr.y.smithi106.stdout:Mar 10 00:46:19 smithi106 ceph-75c37498-de77-11ee-95c7-87774f69a715-mgr-y[25997]: 2024-03-10T00:46:19.178+0000 7f1904d07200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-10T00:46:20.339 INFO:journalctl@ceph.mgr.y.smithi106.stdout:Mar 10 00:46:19 smithi106 ceph-75c37498-de77-11ee-95c7-87774f69a715-mgr-y[25997]: 2024-03-10T00:46:19.964+0000 7f1904d07200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-10T00:46:20.837 INFO:journalctl@ceph.mgr.y.smithi106.stdout:Mar 10 00:46:20 smithi106 ceph-75c37498-de77-11ee-95c7-87774f69a715-mgr-y[25997]: 2024-03-10T00:46:20.526+0000 7f1904d07200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-10T00:46:20.837 INFO:journalctl@ceph.mgr.y.smithi106.stdout:Mar 10 00:46:20 smithi106 ceph-75c37498-de77-11ee-95c7-87774f69a715-mgr-y[25997]: 2024-03-10T00:46:20.621+0000 7f1904d07200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-10T00:46:20.837 INFO:journalctl@ceph.mgr.y.smithi106.stdout:Mar 10 00:46:20 smithi106 ceph-75c37498-de77-11ee-95c7-87774f69a715-mgr-y[25997]: 2024-03-10T00:46:20.837+0000 7f1904d07200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-10T00:46:21.089 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:46:20 smithi106 ceph-mon[25742]: Activating manager daemon y 2024-03-10T00:46:21.089 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:46:20 smithi106 ceph-mon[25742]: mgrmap e2: y(active, starting, since 0.00317356s) 2024-03-10T00:46:21.089 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:46:20 smithi106 ceph-mon[25742]: from='mgr.14100 172.21.15.106:0/2682001965' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-10T00:46:21.090 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:46:20 smithi106 ceph-mon[25742]: from='mgr.14100 172.21.15.106:0/2682001965' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-10T00:46:21.090 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:46:20 smithi106 ceph-mon[25742]: from='mgr.14100 172.21.15.106:0/2682001965' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-10T00:46:21.090 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:46:20 smithi106 ceph-mon[25742]: from='mgr.14100 172.21.15.106:0/2682001965' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-10T00:46:21.090 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:46:20 smithi106 ceph-mon[25742]: from='mgr.14100 172.21.15.106:0/2682001965' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-10T00:46:21.090 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:46:20 smithi106 ceph-mon[25742]: Manager daemon y is now available 2024-03-10T00:46:21.090 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:46:20 smithi106 ceph-mon[25742]: from='mgr.14100 172.21.15.106:0/2682001965' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-10T00:46:21.090 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:46:20 smithi106 ceph-mon[25742]: from='mgr.14100 172.21.15.106:0/2682001965' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-10T00:46:21.090 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:46:20 smithi106 ceph-mon[25742]: from='mgr.14100 172.21.15.106:0/2682001965' entity='mgr.y' 2024-03-10T00:46:21.090 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:46:20 smithi106 ceph-mon[25742]: from='mgr.14100 172.21.15.106:0/2682001965' entity='mgr.y' 2024-03-10T00:46:21.090 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:46:20 smithi106 ceph-mon[25742]: from='mgr.14100 172.21.15.106:0/2682001965' entity='mgr.y' 2024-03-10T00:46:21.940 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:46:21 smithi106 ceph-mon[25742]: from='client.? 172.21.15.106:0/1128560594' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-03-10T00:46:21.940 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:46:21 smithi106 ceph-mon[25742]: mgrmap e3: y(active, since 1.00598s) 2024-03-10T00:46:21.945 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout 2024-03-10T00:46:21.945 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout { 2024-03-10T00:46:21.945 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout "fsid": "75c37498-de77-11ee-95c7-87774f69a715", 2024-03-10T00:46:21.945 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout "health": { 2024-03-10T00:46:21.945 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-03-10T00:46:21.945 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-03-10T00:46:21.945 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-03-10T00:46:21.946 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout }, 2024-03-10T00:46:21.946 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-03-10T00:46:21.946 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-03-10T00:46:21.946 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout 0 2024-03-10T00:46:21.946 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout ], 2024-03-10T00:46:21.946 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-03-10T00:46:21.946 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout "a" 2024-03-10T00:46:21.946 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout ], 2024-03-10T00:46:21.946 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout "quorum_age": 11, 2024-03-10T00:46:21.946 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout "monmap": { 2024-03-10T00:46:21.946 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-10T00:46:21.946 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-03-10T00:46:21.946 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-03-10T00:46:21.946 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout }, 2024-03-10T00:46:21.947 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-03-10T00:46:21.947 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-10T00:46:21.947 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-03-10T00:46:21.947 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-03-10T00:46:21.947 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-03-10T00:46:21.947 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-03-10T00:46:21.947 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-03-10T00:46:21.947 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-03-10T00:46:21.950 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout }, 2024-03-10T00:46:21.950 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-03-10T00:46:21.951 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-03-10T00:46:21.951 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-03-10T00:46:21.951 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-03-10T00:46:21.951 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-03-10T00:46:21.951 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-03-10T00:46:21.951 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-03-10T00:46:21.951 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-03-10T00:46:21.951 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-03-10T00:46:21.951 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout }, 2024-03-10T00:46:21.951 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-03-10T00:46:21.951 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-10T00:46:21.951 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-03-10T00:46:21.951 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-03-10T00:46:21.951 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout }, 2024-03-10T00:46:21.951 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-03-10T00:46:21.952 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout "available": false, 2024-03-10T00:46:21.952 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-03-10T00:46:21.952 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout "modules": [ 2024-03-10T00:46:21.952 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout "iostat", 2024-03-10T00:46:21.952 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout "nfs", 2024-03-10T00:46:21.952 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout "restful" 2024-03-10T00:46:21.952 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout ], 2024-03-10T00:46:21.952 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-10T00:46:21.952 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout }, 2024-03-10T00:46:21.952 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-03-10T00:46:21.952 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-10T00:46:21.952 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout "modified": "2024-03-10T00:46:03.418794+0000", 2024-03-10T00:46:21.952 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-10T00:46:21.952 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout }, 2024-03-10T00:46:21.952 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-03-10T00:46:21.952 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout } 2024-03-10T00:46:21.953 INFO:teuthology.orchestra.run.smithi106.stdout:mgr not available, waiting (3/15)... 2024-03-10T00:46:24.250 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:46:23 smithi106 ceph-mon[25742]: mgrmap e4: y(active, since 2s) 2024-03-10T00:46:25.229 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:46:24 smithi106 ceph-mon[25742]: from='client.? 172.21.15.106:0/3535626286' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-03-10T00:46:25.237 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout 2024-03-10T00:46:25.237 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout { 2024-03-10T00:46:25.237 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout "fsid": "75c37498-de77-11ee-95c7-87774f69a715", 2024-03-10T00:46:25.237 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout "health": { 2024-03-10T00:46:25.237 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-03-10T00:46:25.237 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-03-10T00:46:25.237 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-03-10T00:46:25.238 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout }, 2024-03-10T00:46:25.238 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-03-10T00:46:25.238 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-03-10T00:46:25.238 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout 0 2024-03-10T00:46:25.238 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout ], 2024-03-10T00:46:25.238 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-03-10T00:46:25.238 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout "a" 2024-03-10T00:46:25.238 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout ], 2024-03-10T00:46:25.238 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout "quorum_age": 14, 2024-03-10T00:46:25.238 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout "monmap": { 2024-03-10T00:46:25.238 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-10T00:46:25.238 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-03-10T00:46:25.238 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-03-10T00:46:25.238 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout }, 2024-03-10T00:46:25.239 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-03-10T00:46:25.239 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-10T00:46:25.239 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-03-10T00:46:25.239 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-03-10T00:46:25.239 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-03-10T00:46:25.239 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-03-10T00:46:25.239 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-03-10T00:46:25.239 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-03-10T00:46:25.239 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout }, 2024-03-10T00:46:25.239 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-03-10T00:46:25.239 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-03-10T00:46:25.239 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-03-10T00:46:25.239 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-03-10T00:46:25.239 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-03-10T00:46:25.240 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-03-10T00:46:25.240 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-03-10T00:46:25.240 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-03-10T00:46:25.240 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-03-10T00:46:25.240 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout }, 2024-03-10T00:46:25.240 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-03-10T00:46:25.240 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-10T00:46:25.240 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-03-10T00:46:25.240 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-03-10T00:46:25.240 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout }, 2024-03-10T00:46:25.240 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-03-10T00:46:25.240 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout "available": true, 2024-03-10T00:46:25.240 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-03-10T00:46:25.240 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout "modules": [ 2024-03-10T00:46:25.241 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout "iostat", 2024-03-10T00:46:25.241 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout "nfs", 2024-03-10T00:46:25.241 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout "restful" 2024-03-10T00:46:25.241 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout ], 2024-03-10T00:46:25.241 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-10T00:46:25.241 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout }, 2024-03-10T00:46:25.241 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-03-10T00:46:25.241 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-10T00:46:25.241 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout "modified": "2024-03-10T00:46:03.418794+0000", 2024-03-10T00:46:25.241 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-10T00:46:25.241 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout }, 2024-03-10T00:46:25.241 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-03-10T00:46:25.241 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout } 2024-03-10T00:46:25.241 INFO:teuthology.orchestra.run.smithi106.stdout:mgr is available 2024-03-10T00:46:26.301 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:46:25 smithi106 ceph-mon[25742]: from='client.? 172.21.15.106:0/2882893859' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-03-10T00:46:26.301 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:46:25 smithi106 ceph-mon[25742]: from='client.? 172.21.15.106:0/2882893859' entity='client.admin' cmd='[{"prefix": "config assimilate-conf"}]': finished 2024-03-10T00:46:26.304 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout 2024-03-10T00:46:26.305 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout [global] 2024-03-10T00:46:26.305 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout fsid = 75c37498-de77-11ee-95c7-87774f69a715 2024-03-10T00:46:26.305 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-03-10T00:46:26.305 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-03-10T00:46:26.305 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-03-10T00:46:26.305 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-03-10T00:46:26.305 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout 2024-03-10T00:46:26.305 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout [mgr] 2024-03-10T00:46:26.305 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-03-10T00:46:26.305 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout 2024-03-10T00:46:26.305 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout [osd] 2024-03-10T00:46:26.305 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-03-10T00:46:26.305 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-03-10T00:46:26.305 INFO:teuthology.orchestra.run.smithi106.stdout:Enabling cephadm module... 2024-03-10T00:46:27.339 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:46:27 smithi106 ceph-mon[25742]: from='client.? 172.21.15.106:0/3125141109' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "cephadm"}]: dispatch 2024-03-10T00:46:27.339 INFO:journalctl@ceph.mgr.y.smithi106.stdout:Mar 10 00:46:27 smithi106 ceph-75c37498-de77-11ee-95c7-87774f69a715-mgr-y[25997]: ignoring --setuser ceph since I am not root 2024-03-10T00:46:27.339 INFO:journalctl@ceph.mgr.y.smithi106.stdout:Mar 10 00:46:27 smithi106 ceph-75c37498-de77-11ee-95c7-87774f69a715-mgr-y[25997]: ignoring --setgroup ceph since I am not root 2024-03-10T00:46:27.898 INFO:journalctl@ceph.mgr.y.smithi106.stdout:Mar 10 00:46:27 smithi106 ceph-75c37498-de77-11ee-95c7-87774f69a715-mgr-y[25997]: 2024-03-10T00:46:27.634+0000 7fa3ca584200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-10T00:46:27.899 INFO:journalctl@ceph.mgr.y.smithi106.stdout:Mar 10 00:46:27 smithi106 ceph-75c37498-de77-11ee-95c7-87774f69a715-mgr-y[25997]: 2024-03-10T00:46:27.732+0000 7fa3ca584200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-10T00:46:28.309 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:46:28 smithi106 ceph-mon[25742]: from='client.? 172.21.15.106:0/3125141109' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "cephadm"}]': finished 2024-03-10T00:46:28.309 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:46:28 smithi106 ceph-mon[25742]: mgrmap e5: y(active, since 6s) 2024-03-10T00:46:28.589 INFO:journalctl@ceph.mgr.y.smithi106.stdout:Mar 10 00:46:28 smithi106 ceph-75c37498-de77-11ee-95c7-87774f69a715-mgr-y[25997]: 2024-03-10T00:46:28.430+0000 7fa3ca584200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-10T00:46:28.589 INFO:journalctl@ceph.mgr.y.smithi106.stdout:Mar 10 00:46:28 smithi106 ceph-75c37498-de77-11ee-95c7-87774f69a715-mgr-y[25997]: 2024-03-10T00:46:28.530+0000 7fa3ca584200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-10T00:46:28.937 INFO:journalctl@ceph.mgr.y.smithi106.stdout:Mar 10 00:46:28 smithi106 ceph-75c37498-de77-11ee-95c7-87774f69a715-mgr-y[25997]: 2024-03-10T00:46:28.672+0000 7fa3ca584200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-10T00:46:28.938 INFO:journalctl@ceph.mgr.y.smithi106.stdout:Mar 10 00:46:28 smithi106 ceph-75c37498-de77-11ee-95c7-87774f69a715-mgr-y[25997]: 2024-03-10T00:46:28.769+0000 7fa3ca584200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-10T00:46:28.938 INFO:journalctl@ceph.mgr.y.smithi106.stdout:Mar 10 00:46:28 smithi106 ceph-75c37498-de77-11ee-95c7-87774f69a715-mgr-y[25997]: 2024-03-10T00:46:28.850+0000 7fa3ca584200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-10T00:46:28.941 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout { 2024-03-10T00:46:28.941 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout "epoch": 5, 2024-03-10T00:46:28.941 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout "available": true, 2024-03-10T00:46:28.941 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout "active_name": "y", 2024-03-10T00:46:28.941 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-03-10T00:46:28.941 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout } 2024-03-10T00:46:28.941 INFO:teuthology.orchestra.run.smithi106.stdout:Waiting for the mgr to restart... 2024-03-10T00:46:28.941 INFO:teuthology.orchestra.run.smithi106.stdout:Waiting for mgr epoch 5... 2024-03-10T00:46:29.194 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:46:29 smithi106 ceph-mon[25742]: from='client.? 172.21.15.106:0/3367764765' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-03-10T00:46:29.194 INFO:journalctl@ceph.mgr.y.smithi106.stdout:Mar 10 00:46:29 smithi106 ceph-75c37498-de77-11ee-95c7-87774f69a715-mgr-y[25997]: 2024-03-10T00:46:29.125+0000 7fa3ca584200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-10T00:46:29.589 INFO:journalctl@ceph.mgr.y.smithi106.stdout:Mar 10 00:46:29 smithi106 ceph-75c37498-de77-11ee-95c7-87774f69a715-mgr-y[25997]: 2024-03-10T00:46:29.240+0000 7fa3ca584200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-10T00:46:29.589 INFO:journalctl@ceph.mgr.y.smithi106.stdout:Mar 10 00:46:29 smithi106 ceph-75c37498-de77-11ee-95c7-87774f69a715-mgr-y[25997]: 2024-03-10T00:46:29.328+0000 7fa3ca584200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-10T00:46:29.589 INFO:journalctl@ceph.mgr.y.smithi106.stdout:Mar 10 00:46:29 smithi106 ceph-75c37498-de77-11ee-95c7-87774f69a715-mgr-y[25997]: 2024-03-10T00:46:29.426+0000 7fa3ca584200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-10T00:46:30.089 INFO:journalctl@ceph.mgr.y.smithi106.stdout:Mar 10 00:46:29 smithi106 ceph-75c37498-de77-11ee-95c7-87774f69a715-mgr-y[25997]: 2024-03-10T00:46:29.601+0000 7fa3ca584200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-10T00:46:30.839 INFO:journalctl@ceph.mgr.y.smithi106.stdout:Mar 10 00:46:30 smithi106 ceph-75c37498-de77-11ee-95c7-87774f69a715-mgr-y[25997]: 2024-03-10T00:46:30.551+0000 7fa3ca584200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-10T00:46:30.839 INFO:journalctl@ceph.mgr.y.smithi106.stdout:Mar 10 00:46:30 smithi106 ceph-75c37498-de77-11ee-95c7-87774f69a715-mgr-y[25997]: 2024-03-10T00:46:30.679+0000 7fa3ca584200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-10T00:46:31.296 INFO:journalctl@ceph.mgr.y.smithi106.stdout:Mar 10 00:46:31 smithi106 ceph-75c37498-de77-11ee-95c7-87774f69a715-mgr-y[25997]: 2024-03-10T00:46:31.012+0000 7fa3ca584200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-10T00:46:31.296 INFO:journalctl@ceph.mgr.y.smithi106.stdout:Mar 10 00:46:31 smithi106 ceph-75c37498-de77-11ee-95c7-87774f69a715-mgr-y[25997]: 2024-03-10T00:46:31.101+0000 7fa3ca584200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-10T00:46:31.296 INFO:journalctl@ceph.mgr.y.smithi106.stdout:Mar 10 00:46:31 smithi106 ceph-75c37498-de77-11ee-95c7-87774f69a715-mgr-y[25997]: 2024-03-10T00:46:31.295+0000 7fa3ca584200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-10T00:46:31.589 INFO:journalctl@ceph.mgr.y.smithi106.stdout:Mar 10 00:46:31 smithi106 ceph-75c37498-de77-11ee-95c7-87774f69a715-mgr-y[25997]: 2024-03-10T00:46:31.479+0000 7fa3ca584200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-10T00:46:32.042 INFO:journalctl@ceph.mgr.y.smithi106.stdout:Mar 10 00:46:31 smithi106 ceph-75c37498-de77-11ee-95c7-87774f69a715-mgr-y[25997]: 2024-03-10T00:46:31.763+0000 7fa3ca584200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-10T00:46:32.043 INFO:journalctl@ceph.mgr.y.smithi106.stdout:Mar 10 00:46:32 smithi106 ceph-75c37498-de77-11ee-95c7-87774f69a715-mgr-y[25997]: 2024-03-10T00:46:32.042+0000 7fa3ca584200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-10T00:46:32.339 INFO:journalctl@ceph.mgr.y.smithi106.stdout:Mar 10 00:46:32 smithi106 ceph-75c37498-de77-11ee-95c7-87774f69a715-mgr-y[25997]: 2024-03-10T00:46:32.275+0000 7fa3ca584200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-10T00:46:33.339 INFO:journalctl@ceph.mgr.y.smithi106.stdout:Mar 10 00:46:33 smithi106 ceph-75c37498-de77-11ee-95c7-87774f69a715-mgr-y[25997]: 2024-03-10T00:46:33.066+0000 7fa3ca584200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-10T00:46:33.954 INFO:journalctl@ceph.mgr.y.smithi106.stdout:Mar 10 00:46:33 smithi106 ceph-75c37498-de77-11ee-95c7-87774f69a715-mgr-y[25997]: 2024-03-10T00:46:33.639+0000 7fa3ca584200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-10T00:46:33.954 INFO:journalctl@ceph.mgr.y.smithi106.stdout:Mar 10 00:46:33 smithi106 ceph-75c37498-de77-11ee-95c7-87774f69a715-mgr-y[25997]: 2024-03-10T00:46:33.738+0000 7fa3ca584200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-10T00:46:33.954 INFO:journalctl@ceph.mgr.y.smithi106.stdout:Mar 10 00:46:33 smithi106 ceph-75c37498-de77-11ee-95c7-87774f69a715-mgr-y[25997]: 2024-03-10T00:46:33.954+0000 7fa3ca584200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-10T00:46:34.339 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:46:34 smithi106 ceph-mon[25742]: Active manager daemon y restarted 2024-03-10T00:46:34.339 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:46:34 smithi106 ceph-mon[25742]: Activating manager daemon y 2024-03-10T00:46:34.339 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:46:34 smithi106 ceph-mon[25742]: osdmap e2: 0 total, 0 up, 0 in 2024-03-10T00:46:34.339 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:46:34 smithi106 ceph-mon[25742]: mgrmap e6: y(active, starting, since 0.00332261s) 2024-03-10T00:46:34.339 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:46:34 smithi106 ceph-mon[25742]: from='mgr.14120 172.21.15.106:0/1367199214' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-10T00:46:34.339 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:46:34 smithi106 ceph-mon[25742]: from='mgr.14120 172.21.15.106:0/1367199214' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-10T00:46:34.339 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:46:34 smithi106 ceph-mon[25742]: from='mgr.14120 172.21.15.106:0/1367199214' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-10T00:46:34.340 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:46:34 smithi106 ceph-mon[25742]: from='mgr.14120 172.21.15.106:0/1367199214' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-10T00:46:34.340 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:46:34 smithi106 ceph-mon[25742]: from='mgr.14120 172.21.15.106:0/1367199214' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-10T00:46:34.340 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:46:34 smithi106 ceph-mon[25742]: Manager daemon y is now available 2024-03-10T00:46:34.340 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:46:34 smithi106 ceph-mon[25742]: from='mgr.14120 172.21.15.106:0/1367199214' entity='mgr.y' 2024-03-10T00:46:34.340 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:46:34 smithi106 ceph-mon[25742]: from='mgr.14120 172.21.15.106:0/1367199214' entity='mgr.y' 2024-03-10T00:46:34.340 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:46:34 smithi106 ceph-mon[25742]: from='mgr.14120 172.21.15.106:0/1367199214' entity='mgr.y' 2024-03-10T00:46:34.340 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:46:34 smithi106 ceph-mon[25742]: from='mgr.14120 172.21.15.106:0/1367199214' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-10T00:46:34.340 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:46:34 smithi106 ceph-mon[25742]: from='mgr.14120 172.21.15.106:0/1367199214' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-10T00:46:35.234 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:46:35 smithi106 ceph-mon[25742]: Found migration_current of "None". Setting to last migration. 2024-03-10T00:46:35.234 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:46:35 smithi106 ceph-mon[25742]: from='mgr.14120 172.21.15.106:0/1367199214' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-10T00:46:35.235 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:46:35 smithi106 ceph-mon[25742]: from='mgr.14120 172.21.15.106:0/1367199214' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-10T00:46:35.235 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:46:35 smithi106 ceph-mon[25742]: from='mgr.14120 172.21.15.106:0/1367199214' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-10T00:46:35.235 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:46:35 smithi106 ceph-mon[25742]: from='mgr.14120 172.21.15.106:0/1367199214' entity='mgr.y' 2024-03-10T00:46:35.235 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:46:35 smithi106 ceph-mon[25742]: from='mgr.14120 172.21.15.106:0/1367199214' entity='mgr.y' 2024-03-10T00:46:35.235 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:46:35 smithi106 ceph-mon[25742]: mgrmap e7: y(active, since 1.00557s) 2024-03-10T00:46:35.530 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout { 2024-03-10T00:46:35.530 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 7, 2024-03-10T00:46:35.530 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout "initialized": true 2024-03-10T00:46:35.531 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout } 2024-03-10T00:46:35.531 INFO:teuthology.orchestra.run.smithi106.stdout:mgr epoch 5 is available 2024-03-10T00:46:35.531 INFO:teuthology.orchestra.run.smithi106.stdout:Setting orchestrator backend to cephadm... 2024-03-10T00:46:36.089 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:46:36 smithi106 ceph-mon[25742]: from='client.14124 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-03-10T00:46:36.089 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:46:36 smithi106 ceph-mon[25742]: from='client.14124 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-03-10T00:46:36.089 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:46:36 smithi106 ceph-mon[25742]: from='mgr.14120 172.21.15.106:0/1367199214' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-10T00:46:37.287 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:46:37 smithi106 ceph-mon[25742]: [10/Mar/2024:00:46:35] ENGINE Bus STARTING 2024-03-10T00:46:37.287 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:46:37 smithi106 ceph-mon[25742]: [10/Mar/2024:00:46:35] ENGINE Serving on https://172.21.15.106:7150 2024-03-10T00:46:37.287 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:46:37 smithi106 ceph-mon[25742]: [10/Mar/2024:00:46:35] ENGINE Serving on http://172.21.15.106:8765 2024-03-10T00:46:37.287 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:46:37 smithi106 ceph-mon[25742]: [10/Mar/2024:00:46:35] ENGINE Bus STARTED 2024-03-10T00:46:37.288 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:46:37 smithi106 ceph-mon[25742]: mgrmap e8: y(active, since 2s) 2024-03-10T00:46:37.288 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:46:37 smithi106 ceph-mon[25742]: from='mgr.14120 172.21.15.106:0/1367199214' entity='mgr.y' 2024-03-10T00:46:37.288 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:46:37 smithi106 ceph-mon[25742]: from='mgr.14120 172.21.15.106:0/1367199214' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-10T00:46:37.824 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout value unchanged 2024-03-10T00:46:37.824 INFO:teuthology.orchestra.run.smithi106.stdout:Generating ssh key... 2024-03-10T00:46:38.191 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:46:38 smithi106 ceph-mon[25742]: from='client.14132 -' entity='client.admin' cmd=[{"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}]: dispatch 2024-03-10T00:46:39.027 INFO:journalctl@ceph.mgr.y.smithi106.stdout:Mar 10 00:46:38 smithi106 ceph-75c37498-de77-11ee-95c7-87774f69a715-mgr-y[25997]: Generating public/private rsa key pair. 2024-03-10T00:46:39.028 INFO:journalctl@ceph.mgr.y.smithi106.stdout:Mar 10 00:46:38 smithi106 ceph-75c37498-de77-11ee-95c7-87774f69a715-mgr-y[25997]: Your identification has been saved in /tmp/tmpezp_wdgx/key. 2024-03-10T00:46:39.028 INFO:journalctl@ceph.mgr.y.smithi106.stdout:Mar 10 00:46:38 smithi106 ceph-75c37498-de77-11ee-95c7-87774f69a715-mgr-y[25997]: Your public key has been saved in /tmp/tmpezp_wdgx/key.pub. 2024-03-10T00:46:39.028 INFO:journalctl@ceph.mgr.y.smithi106.stdout:Mar 10 00:46:38 smithi106 ceph-75c37498-de77-11ee-95c7-87774f69a715-mgr-y[25997]: The key fingerprint is: 2024-03-10T00:46:39.028 INFO:journalctl@ceph.mgr.y.smithi106.stdout:Mar 10 00:46:38 smithi106 ceph-75c37498-de77-11ee-95c7-87774f69a715-mgr-y[25997]: SHA256:vToEzqKT1WyAgWPPap7viC/v3Wdf0Y9I2Q9B8M+mbIs ceph-75c37498-de77-11ee-95c7-87774f69a715 2024-03-10T00:46:39.028 INFO:journalctl@ceph.mgr.y.smithi106.stdout:Mar 10 00:46:38 smithi106 ceph-75c37498-de77-11ee-95c7-87774f69a715-mgr-y[25997]: The key's randomart image is: 2024-03-10T00:46:39.028 INFO:journalctl@ceph.mgr.y.smithi106.stdout:Mar 10 00:46:38 smithi106 ceph-75c37498-de77-11ee-95c7-87774f69a715-mgr-y[25997]: +---[RSA 3072]----+ 2024-03-10T00:46:39.028 INFO:journalctl@ceph.mgr.y.smithi106.stdout:Mar 10 00:46:38 smithi106 ceph-75c37498-de77-11ee-95c7-87774f69a715-mgr-y[25997]: | . ... | 2024-03-10T00:46:39.028 INFO:journalctl@ceph.mgr.y.smithi106.stdout:Mar 10 00:46:38 smithi106 ceph-75c37498-de77-11ee-95c7-87774f69a715-mgr-y[25997]: |o.. o | 2024-03-10T00:46:39.028 INFO:journalctl@ceph.mgr.y.smithi106.stdout:Mar 10 00:46:38 smithi106 ceph-75c37498-de77-11ee-95c7-87774f69a715-mgr-y[25997]: |..oo o | 2024-03-10T00:46:39.028 INFO:journalctl@ceph.mgr.y.smithi106.stdout:Mar 10 00:46:38 smithi106 ceph-75c37498-de77-11ee-95c7-87774f69a715-mgr-y[25997]: | .o. . . + + | 2024-03-10T00:46:39.028 INFO:journalctl@ceph.mgr.y.smithi106.stdout:Mar 10 00:46:38 smithi106 ceph-75c37498-de77-11ee-95c7-87774f69a715-mgr-y[25997]: | . * .S . + + +| 2024-03-10T00:46:39.028 INFO:journalctl@ceph.mgr.y.smithi106.stdout:Mar 10 00:46:38 smithi106 ceph-75c37498-de77-11ee-95c7-87774f69a715-mgr-y[25997]: | o o * . o + B | 2024-03-10T00:46:39.028 INFO:journalctl@ceph.mgr.y.smithi106.stdout:Mar 10 00:46:38 smithi106 ceph-75c37498-de77-11ee-95c7-87774f69a715-mgr-y[25997]: |o .+ o . . o = o| 2024-03-10T00:46:39.028 INFO:journalctl@ceph.mgr.y.smithi106.stdout:Mar 10 00:46:38 smithi106 ceph-75c37498-de77-11ee-95c7-87774f69a715-mgr-y[25997]: |oo=. . +. . o . | 2024-03-10T00:46:39.029 INFO:journalctl@ceph.mgr.y.smithi106.stdout:Mar 10 00:46:38 smithi106 ceph-75c37498-de77-11ee-95c7-87774f69a715-mgr-y[25997]: |.**+. .o.o. E . | 2024-03-10T00:46:39.029 INFO:journalctl@ceph.mgr.y.smithi106.stdout:Mar 10 00:46:38 smithi106 ceph-75c37498-de77-11ee-95c7-87774f69a715-mgr-y[25997]: +----[SHA256]-----+ 2024-03-10T00:46:39.302 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:46:39 smithi106 ceph-mon[25742]: from='client.14134 -' entity='client.admin' cmd=[{"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}]: dispatch 2024-03-10T00:46:39.303 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:46:39 smithi106 ceph-mon[25742]: from='mgr.14120 172.21.15.106:0/1367199214' entity='mgr.y' 2024-03-10T00:46:39.303 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:46:39 smithi106 ceph-mon[25742]: from='mgr.14120 172.21.15.106:0/1367199214' entity='mgr.y' 2024-03-10T00:46:40.333 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:46:40 smithi106 ceph-mon[25742]: from='client.14136 -' entity='client.admin' cmd=[{"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}]: dispatch 2024-03-10T00:46:40.333 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:46:40 smithi106 ceph-mon[25742]: Generating ssh key... 2024-03-10T00:46:40.561 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDZjzaDWVwLqYhRDUUTLmtOMtOSY67TTwHRtPKq79TPb/ll/W9yqwdQ+00evlxbue2VIV/GUn6zsbtflt9juaxj9e6tE7+Pz/PTQaaYvN85cmGFvpXW7CNo0U7Vwd4w4ThEAvvrlxeeZdkUu06zS7pISAXAmj5FeTe/9SQAOfilS+EF5TKAqX/OG85iWiNsLYofr+rHDrIEKLwRGPwcJL6lbiVnjVthSpnrrhKCZlRWzOcLMo7O/eMMSd69pzLq47CC05jJnkAtd3L1vLGrujD6QDIyng6OmmtZDtJLjqi1RuaUUmtv+5mSiew8wD1NEyWXClAhDTX3ncIoVM5jiaL1+31MXj9yTsu91DYvUyaxb5JfoHnTTmLUU7QiWgQzW7QMlkaBF6vwG08E4vfbYPaWswrlc8P5FkUgNF+6wypTUd/b/GJ0ew46/rvvRXD3MSgzJptk88i3OmP1xnt0mybEHGDe8nfuArUkWa83cSigRdspmQ/XYkhgm6B0V9hr6Us= ceph-75c37498-de77-11ee-95c7-87774f69a715 2024-03-10T00:46:40.561 INFO:teuthology.orchestra.run.smithi106.stdout:Wrote public SSH key to /home/ubuntu/cephtest/ceph.pub 2024-03-10T00:46:40.561 INFO:teuthology.orchestra.run.smithi106.stdout:Adding key to root@localhost authorized_keys... 2024-03-10T00:46:40.561 INFO:teuthology.orchestra.run.smithi106.stdout:Adding host smithi106... 2024-03-10T00:46:41.280 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:46:41 smithi106 ceph-mon[25742]: from='client.14138 -' entity='client.admin' cmd=[{"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}]: dispatch 2024-03-10T00:46:42.458 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:46:42 smithi106 ceph-mon[25742]: from='client.14140 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi106", "addr": "172.21.15.106", "target": ["mon-mgr", ""]}]: dispatch 2024-03-10T00:46:42.459 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:46:42 smithi106 ceph-mon[25742]: Deploying cephadm binary to smithi106 2024-03-10T00:46:46.143 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout Added host 'smithi106' with addr '172.21.15.106' 2024-03-10T00:46:46.143 INFO:teuthology.orchestra.run.smithi106.stdout:Deploying unmanaged mon service... 2024-03-10T00:46:46.789 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:46:46 smithi106 ceph-mon[25742]: from='mgr.14120 172.21.15.106:0/1367199214' entity='mgr.y' 2024-03-10T00:46:46.789 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:46:46 smithi106 ceph-mon[25742]: Added host smithi106 2024-03-10T00:46:46.789 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:46:46 smithi106 ceph-mon[25742]: from='mgr.14120 172.21.15.106:0/1367199214' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-10T00:46:48.036 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:46:47 smithi106 ceph-mon[25742]: from='client.14142 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2024-03-10T00:46:48.037 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:46:47 smithi106 ceph-mon[25742]: Saving service mon spec with placement count:5 2024-03-10T00:46:48.037 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:46:47 smithi106 ceph-mon[25742]: from='mgr.14120 172.21.15.106:0/1367199214' entity='mgr.y' 2024-03-10T00:46:48.491 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout Scheduled mon update... 2024-03-10T00:46:48.491 INFO:teuthology.orchestra.run.smithi106.stdout:Deploying unmanaged mgr service... 2024-03-10T00:46:49.486 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:46:49 smithi106 ceph-mon[25742]: from='mgr.14120 172.21.15.106:0/1367199214' entity='mgr.y' 2024-03-10T00:46:49.486 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:46:49 smithi106 ceph-mon[25742]: from='mgr.14120 172.21.15.106:0/1367199214' entity='mgr.y' 2024-03-10T00:46:49.772 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout Scheduled mgr update... 2024-03-10T00:46:50.472 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:46:50 smithi106 ceph-mon[25742]: from='client.14144 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2024-03-10T00:46:50.472 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:46:50 smithi106 ceph-mon[25742]: Saving service mgr spec with placement count:2 2024-03-10T00:46:50.472 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:46:50 smithi106 ceph-mon[25742]: from='mgr.14120 172.21.15.106:0/1367199214' entity='mgr.y' 2024-03-10T00:46:51.589 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:46:51 smithi106 ceph-mon[25742]: from='client.? 172.21.15.106:0/1985763529' entity='client.admin' 2024-03-10T00:46:52.461 INFO:teuthology.orchestra.run.smithi106.stdout:Enabling the dashboard module... 2024-03-10T00:46:53.089 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:46:52 smithi106 ceph-mon[25742]: from='client.? 172.21.15.106:0/2616688991' entity='client.admin' 2024-03-10T00:46:53.089 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:46:52 smithi106 ceph-mon[25742]: from='mgr.14120 172.21.15.106:0/1367199214' entity='mgr.y' 2024-03-10T00:46:53.089 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:46:52 smithi106 ceph-mon[25742]: from='mgr.14120 172.21.15.106:0/1367199214' entity='mgr.y' 2024-03-10T00:46:53.089 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:46:52 smithi106 ceph-mon[25742]: from='mgr.14120 172.21.15.106:0/1367199214' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi106", "name": "osd_memory_target"}]: dispatch 2024-03-10T00:46:53.089 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:46:52 smithi106 ceph-mon[25742]: from='mgr.14120 172.21.15.106:0/1367199214' entity='mgr.y' 2024-03-10T00:46:53.089 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:46:52 smithi106 ceph-mon[25742]: from='mgr.14120 172.21.15.106:0/1367199214' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.agent.smithi106", "caps": []}]: dispatch 2024-03-10T00:46:53.089 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:46:52 smithi106 ceph-mon[25742]: from='mgr.14120 172.21.15.106:0/1367199214' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.agent.smithi106", "caps": []}]': finished 2024-03-10T00:46:54.089 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:46:53 smithi106 ceph-mon[25742]: Deploying daemon agent.smithi106 on smithi106 2024-03-10T00:46:54.089 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:46:53 smithi106 ceph-mon[25742]: from='client.? 172.21.15.106:0/1811835768' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "dashboard"}]: dispatch 2024-03-10T00:46:54.089 INFO:journalctl@ceph.mgr.y.smithi106.stdout:Mar 10 00:46:53 smithi106 ceph-75c37498-de77-11ee-95c7-87774f69a715-mgr-y[25997]: ignoring --setuser ceph since I am not root 2024-03-10T00:46:54.089 INFO:journalctl@ceph.mgr.y.smithi106.stdout:Mar 10 00:46:53 smithi106 ceph-75c37498-de77-11ee-95c7-87774f69a715-mgr-y[25997]: ignoring --setgroup ceph since I am not root 2024-03-10T00:46:54.505 INFO:journalctl@ceph.mgr.y.smithi106.stdout:Mar 10 00:46:54 smithi106 ceph-75c37498-de77-11ee-95c7-87774f69a715-mgr-y[25997]: 2024-03-10T00:46:54.408+0000 7feef2d46200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-10T00:46:54.505 INFO:journalctl@ceph.mgr.y.smithi106.stdout:Mar 10 00:46:54 smithi106 ceph-75c37498-de77-11ee-95c7-87774f69a715-mgr-y[25997]: 2024-03-10T00:46:54.506+0000 7feef2d46200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-10T00:46:55.089 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:46:54 smithi106 ceph-mon[25742]: from='client.? 172.21.15.106:0/1811835768' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "dashboard"}]': finished 2024-03-10T00:46:55.089 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:46:54 smithi106 ceph-mon[25742]: mgrmap e9: y(active, since 19s) 2024-03-10T00:46:55.502 INFO:journalctl@ceph.mgr.y.smithi106.stdout:Mar 10 00:46:55 smithi106 ceph-75c37498-de77-11ee-95c7-87774f69a715-mgr-y[25997]: 2024-03-10T00:46:55.165+0000 7feef2d46200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-10T00:46:55.502 INFO:journalctl@ceph.mgr.y.smithi106.stdout:Mar 10 00:46:55 smithi106 ceph-75c37498-de77-11ee-95c7-87774f69a715-mgr-y[25997]: 2024-03-10T00:46:55.262+0000 7feef2d46200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-10T00:46:55.503 INFO:journalctl@ceph.mgr.y.smithi106.stdout:Mar 10 00:46:55 smithi106 ceph-75c37498-de77-11ee-95c7-87774f69a715-mgr-y[25997]: 2024-03-10T00:46:55.405+0000 7feef2d46200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-10T00:46:55.503 INFO:journalctl@ceph.mgr.y.smithi106.stdout:Mar 10 00:46:55 smithi106 ceph-75c37498-de77-11ee-95c7-87774f69a715-mgr-y[25997]: 2024-03-10T00:46:55.503+0000 7feef2d46200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-10T00:46:55.750 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout { 2024-03-10T00:46:55.750 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout "epoch": 9, 2024-03-10T00:46:55.750 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout "available": true, 2024-03-10T00:46:55.750 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout "active_name": "y", 2024-03-10T00:46:55.750 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-03-10T00:46:55.750 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout } 2024-03-10T00:46:55.750 INFO:teuthology.orchestra.run.smithi106.stdout:Waiting for the mgr to restart... 2024-03-10T00:46:55.750 INFO:teuthology.orchestra.run.smithi106.stdout:Waiting for mgr epoch 9... 2024-03-10T00:46:55.839 INFO:journalctl@ceph.mgr.y.smithi106.stdout:Mar 10 00:46:55 smithi106 ceph-75c37498-de77-11ee-95c7-87774f69a715-mgr-y[25997]: 2024-03-10T00:46:55.588+0000 7feef2d46200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-10T00:46:56.141 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:46:55 smithi106 ceph-mon[25742]: from='client.? 172.21.15.106:0/1579352218' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-03-10T00:46:56.142 INFO:journalctl@ceph.mgr.y.smithi106.stdout:Mar 10 00:46:55 smithi106 ceph-75c37498-de77-11ee-95c7-87774f69a715-mgr-y[25997]: 2024-03-10T00:46:55.862+0000 7feef2d46200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-10T00:46:56.142 INFO:journalctl@ceph.mgr.y.smithi106.stdout:Mar 10 00:46:55 smithi106 ceph-75c37498-de77-11ee-95c7-87774f69a715-mgr-y[25997]: 2024-03-10T00:46:55.973+0000 7feef2d46200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-10T00:46:56.142 INFO:journalctl@ceph.mgr.y.smithi106.stdout:Mar 10 00:46:56 smithi106 ceph-75c37498-de77-11ee-95c7-87774f69a715-mgr-y[25997]: 2024-03-10T00:46:56.065+0000 7feef2d46200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-10T00:46:56.589 INFO:journalctl@ceph.mgr.y.smithi106.stdout:Mar 10 00:46:56 smithi106 ceph-75c37498-de77-11ee-95c7-87774f69a715-mgr-y[25997]: 2024-03-10T00:46:56.162+0000 7feef2d46200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-10T00:46:56.589 INFO:journalctl@ceph.mgr.y.smithi106.stdout:Mar 10 00:46:56 smithi106 ceph-75c37498-de77-11ee-95c7-87774f69a715-mgr-y[25997]: 2024-03-10T00:46:56.343+0000 7feef2d46200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-10T00:46:57.589 INFO:journalctl@ceph.mgr.y.smithi106.stdout:Mar 10 00:46:57 smithi106 ceph-75c37498-de77-11ee-95c7-87774f69a715-mgr-y[25997]: 2024-03-10T00:46:57.300+0000 7feef2d46200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-10T00:46:57.589 INFO:journalctl@ceph.mgr.y.smithi106.stdout:Mar 10 00:46:57 smithi106 ceph-75c37498-de77-11ee-95c7-87774f69a715-mgr-y[25997]: 2024-03-10T00:46:57.430+0000 7feef2d46200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-10T00:46:58.048 INFO:journalctl@ceph.mgr.y.smithi106.stdout:Mar 10 00:46:57 smithi106 ceph-75c37498-de77-11ee-95c7-87774f69a715-mgr-y[25997]: 2024-03-10T00:46:57.763+0000 7feef2d46200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-10T00:46:58.049 INFO:journalctl@ceph.mgr.y.smithi106.stdout:Mar 10 00:46:57 smithi106 ceph-75c37498-de77-11ee-95c7-87774f69a715-mgr-y[25997]: 2024-03-10T00:46:57.854+0000 7feef2d46200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-10T00:46:58.049 INFO:journalctl@ceph.mgr.y.smithi106.stdout:Mar 10 00:46:58 smithi106 ceph-75c37498-de77-11ee-95c7-87774f69a715-mgr-y[25997]: 2024-03-10T00:46:58.049+0000 7feef2d46200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-10T00:46:58.339 INFO:journalctl@ceph.mgr.y.smithi106.stdout:Mar 10 00:46:58 smithi106 ceph-75c37498-de77-11ee-95c7-87774f69a715-mgr-y[25997]: 2024-03-10T00:46:58.234+0000 7feef2d46200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-10T00:46:58.792 INFO:journalctl@ceph.mgr.y.smithi106.stdout:Mar 10 00:46:58 smithi106 ceph-75c37498-de77-11ee-95c7-87774f69a715-mgr-y[25997]: 2024-03-10T00:46:58.523+0000 7feef2d46200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-10T00:46:58.792 INFO:journalctl@ceph.mgr.y.smithi106.stdout:Mar 10 00:46:58 smithi106 ceph-75c37498-de77-11ee-95c7-87774f69a715-mgr-y[25997]: 2024-03-10T00:46:58.793+0000 7feef2d46200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-10T00:46:59.089 INFO:journalctl@ceph.mgr.y.smithi106.stdout:Mar 10 00:46:59 smithi106 ceph-75c37498-de77-11ee-95c7-87774f69a715-mgr-y[25997]: 2024-03-10T00:46:59.029+0000 7feef2d46200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-10T00:47:00.089 INFO:journalctl@ceph.mgr.y.smithi106.stdout:Mar 10 00:46:59 smithi106 ceph-75c37498-de77-11ee-95c7-87774f69a715-mgr-y[25997]: 2024-03-10T00:46:59.831+0000 7feef2d46200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-10T00:47:00.718 INFO:journalctl@ceph.mgr.y.smithi106.stdout:Mar 10 00:47:00 smithi106 ceph-75c37498-de77-11ee-95c7-87774f69a715-mgr-y[25997]: 2024-03-10T00:47:00.398+0000 7feef2d46200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-10T00:47:00.718 INFO:journalctl@ceph.mgr.y.smithi106.stdout:Mar 10 00:47:00 smithi106 ceph-75c37498-de77-11ee-95c7-87774f69a715-mgr-y[25997]: 2024-03-10T00:47:00.498+0000 7feef2d46200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-10T00:47:00.718 INFO:journalctl@ceph.mgr.y.smithi106.stdout:Mar 10 00:47:00 smithi106 ceph-75c37498-de77-11ee-95c7-87774f69a715-mgr-y[25997]: 2024-03-10T00:47:00.719+0000 7feef2d46200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-10T00:47:01.089 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:47:00 smithi106 ceph-mon[25742]: Active manager daemon y restarted 2024-03-10T00:47:01.089 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:47:00 smithi106 ceph-mon[25742]: Activating manager daemon y 2024-03-10T00:47:01.089 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:47:00 smithi106 ceph-mon[25742]: osdmap e3: 0 total, 0 up, 0 in 2024-03-10T00:47:01.089 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:47:00 smithi106 ceph-mon[25742]: mgrmap e10: y(active, starting, since 0.00338814s) 2024-03-10T00:47:01.089 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:47:00 smithi106 ceph-mon[25742]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-10T00:47:01.089 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:47:00 smithi106 ceph-mon[25742]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-10T00:47:01.089 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:47:00 smithi106 ceph-mon[25742]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-10T00:47:01.089 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:47:00 smithi106 ceph-mon[25742]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-10T00:47:01.089 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:47:00 smithi106 ceph-mon[25742]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-10T00:47:01.090 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:47:00 smithi106 ceph-mon[25742]: Manager daemon y is now available 2024-03-10T00:47:01.090 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:47:00 smithi106 ceph-mon[25742]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:47:01.090 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:47:00 smithi106 ceph-mon[25742]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-10T00:47:01.090 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:47:00 smithi106 ceph-mon[25742]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-10T00:47:02.089 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:47:01 smithi106 ceph-mon[25742]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-10T00:47:02.089 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:47:01 smithi106 ceph-mon[25742]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-10T00:47:02.089 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:47:01 smithi106 ceph-mon[25742]: mgrmap e11: y(active, since 1.00566s) 2024-03-10T00:47:02.482 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout { 2024-03-10T00:47:02.483 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 11, 2024-03-10T00:47:02.483 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout "initialized": true 2024-03-10T00:47:02.483 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout } 2024-03-10T00:47:02.483 INFO:teuthology.orchestra.run.smithi106.stdout:mgr epoch 9 is available 2024-03-10T00:47:02.483 INFO:teuthology.orchestra.run.smithi106.stdout:Generating a dashboard self-signed certificate... 2024-03-10T00:47:02.931 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:47:02 smithi106 ceph-mon[25742]: [10/Mar/2024:00:47:01] ENGINE Bus STARTING 2024-03-10T00:47:02.932 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:47:02 smithi106 ceph-mon[25742]: [10/Mar/2024:00:47:01] ENGINE Serving on https://172.21.15.106:7150 2024-03-10T00:47:02.932 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:47:02 smithi106 ceph-mon[25742]: from='client.14156 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-03-10T00:47:02.932 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:47:02 smithi106 ceph-mon[25742]: from='client.14156 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-03-10T00:47:02.932 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:47:02 smithi106 ceph-mon[25742]: [10/Mar/2024:00:47:01] ENGINE Serving on http://172.21.15.106:8765 2024-03-10T00:47:02.932 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:47:02 smithi106 ceph-mon[25742]: [10/Mar/2024:00:47:01] ENGINE Bus STARTED 2024-03-10T00:47:03.838 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:47:03 smithi106 ceph-mon[25742]: mgrmap e12: y(active, since 2s) 2024-03-10T00:47:03.839 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:47:03 smithi106 ceph-mon[25742]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:47:03.839 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:47:03 smithi106 ceph-mon[25742]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi106", "name": "osd_memory_target"}]: dispatch 2024-03-10T00:47:03.839 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:47:03 smithi106 ceph-mon[25742]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:47:03.839 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:47:03 smithi106 ceph-mon[25742]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.agent.smithi106", "caps": []}]: dispatch 2024-03-10T00:47:03.839 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:47:03 smithi106 ceph-mon[25742]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:47:03.839 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:47:03 smithi106 ceph-mon[25742]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:47:04.131 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout Self-signed certificate created 2024-03-10T00:47:04.131 INFO:teuthology.orchestra.run.smithi106.stdout:Creating initial admin user... 2024-03-10T00:47:05.089 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:47:04 smithi106 ceph-mon[25742]: Deploying daemon agent.smithi106 on smithi106 2024-03-10T00:47:05.089 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:47:04 smithi106 ceph-mon[25742]: from='client.14164 -' entity='client.admin' cmd=[{"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}]: dispatch 2024-03-10T00:47:05.564 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout {"username": "admin", "password": "$2b$12$IucVuFcHYHA1pMa0BTQMy./PSyWfl0pLZw4ds.A7AbbtlE/Y76.9m", "roles": ["administrator"], "name": null, "email": null, "lastUpdate": 1710031624, "enabled": true, "pwdExpirationDate": null, "pwdUpdateRequired": true} 2024-03-10T00:47:05.564 INFO:teuthology.orchestra.run.smithi106.stdout:Fetching dashboard port number... 2024-03-10T00:47:05.811 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:47:05 smithi106 ceph-mon[25742]: from='client.14166 -' entity='client.admin' cmd=[{"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}]: dispatch 2024-03-10T00:47:05.812 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:47:05 smithi106 ceph-mon[25742]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:47:06.835 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout 8443 2024-03-10T00:47:06.844 INFO:teuthology.orchestra.run.smithi106.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-03-10T00:47:06.844 INFO:teuthology.orchestra.run.smithi106.stdout:systemctl: stdout disabled 2024-03-10T00:47:06.851 INFO:teuthology.orchestra.run.smithi106.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-03-10T00:47:06.851 INFO:teuthology.orchestra.run.smithi106.stdout:systemctl: stdout inactive 2024-03-10T00:47:06.851 INFO:teuthology.orchestra.run.smithi106.stdout:firewalld.service is not enabled 2024-03-10T00:47:06.851 INFO:teuthology.orchestra.run.smithi106.stdout:Not possible to open ports <[8443]>. firewalld.service is not available 2024-03-10T00:47:06.852 INFO:teuthology.orchestra.run.smithi106.stdout:Ceph Dashboard is now available at: 2024-03-10T00:47:06.853 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-03-10T00:47:06.853 INFO:teuthology.orchestra.run.smithi106.stdout: URL: https://smithi106.front.sepia.ceph.com:8443/ 2024-03-10T00:47:06.853 INFO:teuthology.orchestra.run.smithi106.stdout: User: admin 2024-03-10T00:47:06.853 INFO:teuthology.orchestra.run.smithi106.stdout: Password: j2ptfb3d3j 2024-03-10T00:47:06.853 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-03-10T00:47:06.853 INFO:teuthology.orchestra.run.smithi106.stdout:Saving cluster configuration to /var/lib/ceph/75c37498-de77-11ee-95c7-87774f69a715/config directory 2024-03-10T00:47:06.853 INFO:teuthology.orchestra.run.smithi106.stdout:Enabling autotune for osd_memory_target 2024-03-10T00:47:07.085 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:47:06 smithi106 ceph-mon[25742]: from='client.? 172.21.15.106:0/1945722606' entity='client.admin' cmd=[{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]: dispatch 2024-03-10T00:47:08.089 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:47:07 smithi106 ceph-mon[25742]: mgrmap e13: y(active, since 6s) 2024-03-10T00:47:09.382 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr set mgr/dashboard/cluster/status 2024-03-10T00:47:09.382 INFO:teuthology.orchestra.run.smithi106.stdout:You can access the Ceph CLI as following in case of multi-cluster or non-default config: 2024-03-10T00:47:09.382 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-03-10T00:47:09.382 INFO:teuthology.orchestra.run.smithi106.stdout: sudo /home/ubuntu/cephtest/cephadm shell --fsid 75c37498-de77-11ee-95c7-87774f69a715 -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring 2024-03-10T00:47:09.382 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-03-10T00:47:09.383 INFO:teuthology.orchestra.run.smithi106.stdout:Or, if you are only running a single cluster on this host: 2024-03-10T00:47:09.383 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-03-10T00:47:09.383 INFO:teuthology.orchestra.run.smithi106.stdout: sudo /home/ubuntu/cephtest/cephadm shell 2024-03-10T00:47:09.383 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-03-10T00:47:09.383 INFO:teuthology.orchestra.run.smithi106.stdout:Please consider enabling telemetry to help improve Ceph: 2024-03-10T00:47:09.383 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-03-10T00:47:09.383 INFO:teuthology.orchestra.run.smithi106.stdout: ceph telemetry on 2024-03-10T00:47:09.383 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-03-10T00:47:09.383 INFO:teuthology.orchestra.run.smithi106.stdout:For more information see: 2024-03-10T00:47:09.383 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-03-10T00:47:09.383 INFO:teuthology.orchestra.run.smithi106.stdout: https://docs.ceph.com/en/latest/mgr/telemetry/ 2024-03-10T00:47:09.383 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-03-10T00:47:09.383 INFO:teuthology.orchestra.run.smithi106.stdout:Bootstrap complete. 2024-03-10T00:47:09.423 INFO:tasks.cephadm:Fetching config... 2024-03-10T00:47:09.424 DEBUG:teuthology.orchestra.run.smithi106:> set -ex 2024-03-10T00:47:09.424 DEBUG:teuthology.orchestra.run.smithi106:> dd if=/etc/ceph/ceph.conf of=/dev/stdout 2024-03-10T00:47:09.439 INFO:tasks.cephadm:Fetching client.admin keyring... 2024-03-10T00:47:09.439 DEBUG:teuthology.orchestra.run.smithi106:> set -ex 2024-03-10T00:47:09.440 DEBUG:teuthology.orchestra.run.smithi106:> dd if=/etc/ceph/ceph.client.admin.keyring of=/dev/stdout 2024-03-10T00:47:09.495 INFO:tasks.cephadm:Fetching mon keyring... 2024-03-10T00:47:09.495 DEBUG:teuthology.orchestra.run.smithi106:> set -ex 2024-03-10T00:47:09.495 DEBUG:teuthology.orchestra.run.smithi106:> sudo dd if=/var/lib/ceph/75c37498-de77-11ee-95c7-87774f69a715/mon.a/keyring of=/dev/stdout 2024-03-10T00:47:09.679 INFO:tasks.cephadm:Fetching pub ssh key... 2024-03-10T00:47:09.680 DEBUG:teuthology.orchestra.run.smithi106:> set -ex 2024-03-10T00:47:09.680 DEBUG:teuthology.orchestra.run.smithi106:> dd if=/home/ubuntu/cephtest/ceph.pub of=/dev/stdout 2024-03-10T00:47:09.734 INFO:tasks.cephadm:Installing pub ssh key for root users... 2024-03-10T00:47:09.734 DEBUG:teuthology.orchestra.run.smithi106:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDZjzaDWVwLqYhRDUUTLmtOMtOSY67TTwHRtPKq79TPb/ll/W9yqwdQ+00evlxbue2VIV/GUn6zsbtflt9juaxj9e6tE7+Pz/PTQaaYvN85cmGFvpXW7CNo0U7Vwd4w4ThEAvvrlxeeZdkUu06zS7pISAXAmj5FeTe/9SQAOfilS+EF5TKAqX/OG85iWiNsLYofr+rHDrIEKLwRGPwcJL6lbiVnjVthSpnrrhKCZlRWzOcLMo7O/eMMSd69pzLq47CC05jJnkAtd3L1vLGrujD6QDIyng6OmmtZDtJLjqi1RuaUUmtv+5mSiew8wD1NEyWXClAhDTX3ncIoVM5jiaL1+31MXj9yTsu91DYvUyaxb5JfoHnTTmLUU7QiWgQzW7QMlkaBF6vwG08E4vfbYPaWswrlc8P5FkUgNF+6wypTUd/b/GJ0ew46/rvvRXD3MSgzJptk88i3OmP1xnt0mybEHGDe8nfuArUkWa83cSigRdspmQ/XYkhgm6B0V9hr6Us= ceph-75c37498-de77-11ee-95c7-87774f69a715' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-03-10T00:47:09.892 INFO:teuthology.orchestra.run.smithi106.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDZjzaDWVwLqYhRDUUTLmtOMtOSY67TTwHRtPKq79TPb/ll/W9yqwdQ+00evlxbue2VIV/GUn6zsbtflt9juaxj9e6tE7+Pz/PTQaaYvN85cmGFvpXW7CNo0U7Vwd4w4ThEAvvrlxeeZdkUu06zS7pISAXAmj5FeTe/9SQAOfilS+EF5TKAqX/OG85iWiNsLYofr+rHDrIEKLwRGPwcJL6lbiVnjVthSpnrrhKCZlRWzOcLMo7O/eMMSd69pzLq47CC05jJnkAtd3L1vLGrujD6QDIyng6OmmtZDtJLjqi1RuaUUmtv+5mSiew8wD1NEyWXClAhDTX3ncIoVM5jiaL1+31MXj9yTsu91DYvUyaxb5JfoHnTTmLUU7QiWgQzW7QMlkaBF6vwG08E4vfbYPaWswrlc8P5FkUgNF+6wypTUd/b/GJ0ew46/rvvRXD3MSgzJptk88i3OmP1xnt0mybEHGDe8nfuArUkWa83cSigRdspmQ/XYkhgm6B0V9hr6Us= ceph-75c37498-de77-11ee-95c7-87774f69a715 2024-03-10T00:47:09.907 DEBUG:teuthology.orchestra.run.smithi170:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDZjzaDWVwLqYhRDUUTLmtOMtOSY67TTwHRtPKq79TPb/ll/W9yqwdQ+00evlxbue2VIV/GUn6zsbtflt9juaxj9e6tE7+Pz/PTQaaYvN85cmGFvpXW7CNo0U7Vwd4w4ThEAvvrlxeeZdkUu06zS7pISAXAmj5FeTe/9SQAOfilS+EF5TKAqX/OG85iWiNsLYofr+rHDrIEKLwRGPwcJL6lbiVnjVthSpnrrhKCZlRWzOcLMo7O/eMMSd69pzLq47CC05jJnkAtd3L1vLGrujD6QDIyng6OmmtZDtJLjqi1RuaUUmtv+5mSiew8wD1NEyWXClAhDTX3ncIoVM5jiaL1+31MXj9yTsu91DYvUyaxb5JfoHnTTmLUU7QiWgQzW7QMlkaBF6vwG08E4vfbYPaWswrlc8P5FkUgNF+6wypTUd/b/GJ0ew46/rvvRXD3MSgzJptk88i3OmP1xnt0mybEHGDe8nfuArUkWa83cSigRdspmQ/XYkhgm6B0V9hr6Us= ceph-75c37498-de77-11ee-95c7-87774f69a715' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-03-10T00:47:09.930 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:47:09 smithi106 ceph-mon[25742]: from='client.? 172.21.15.106:0/3059978068' entity='client.admin' 2024-03-10T00:47:09.953 INFO:teuthology.orchestra.run.smithi170.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDZjzaDWVwLqYhRDUUTLmtOMtOSY67TTwHRtPKq79TPb/ll/W9yqwdQ+00evlxbue2VIV/GUn6zsbtflt9juaxj9e6tE7+Pz/PTQaaYvN85cmGFvpXW7CNo0U7Vwd4w4ThEAvvrlxeeZdkUu06zS7pISAXAmj5FeTe/9SQAOfilS+EF5TKAqX/OG85iWiNsLYofr+rHDrIEKLwRGPwcJL6lbiVnjVthSpnrrhKCZlRWzOcLMo7O/eMMSd69pzLq47CC05jJnkAtd3L1vLGrujD6QDIyng6OmmtZDtJLjqi1RuaUUmtv+5mSiew8wD1NEyWXClAhDTX3ncIoVM5jiaL1+31MXj9yTsu91DYvUyaxb5JfoHnTTmLUU7QiWgQzW7QMlkaBF6vwG08E4vfbYPaWswrlc8P5FkUgNF+6wypTUd/b/GJ0ew46/rvvRXD3MSgzJptk88i3OmP1xnt0mybEHGDe8nfuArUkWa83cSigRdspmQ/XYkhgm6B0V9hr6Us= ceph-75c37498-de77-11ee-95c7-87774f69a715 2024-03-10T00:47:09.968 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:831bc188fa69f143bbba69ac76119e784500e8e3 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 75c37498-de77-11ee-95c7-87774f69a715 -- ceph config set mgr mgr/cephadm/allow_ptrace true 2024-03-10T00:47:10.250 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/75c37498-de77-11ee-95c7-87774f69a715/mon.a/config 2024-03-10T00:47:11.089 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:47:10 smithi106 ceph-mon[25742]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:47:11.089 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:47:10 smithi106 ceph-mon[25742]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:47:11.089 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:47:10 smithi106 ceph-mon[25742]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:47:11.090 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:47:10 smithi106 ceph-mon[25742]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-10T00:47:11.090 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:47:10 smithi106 ceph-mon[25742]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:47:11.090 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:47:10 smithi106 ceph-mon[25742]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:47:11.090 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:47:10 smithi106 ceph-mon[25742]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:47:11.090 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:47:10 smithi106 ceph-mon[25742]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:47:11.090 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:47:10 smithi106 ceph-mon[25742]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:47:11.090 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:47:10 smithi106 ceph-mon[25742]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-10T00:47:11.090 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:47:10 smithi106 ceph-mon[25742]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:47:11.090 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:47:10 smithi106 ceph-mon[25742]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:47:12.339 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:47:12 smithi106 ceph-mon[25742]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:47:12.339 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:47:12 smithi106 ceph-mon[25742]: from='client.? 172.21.15.106:0/2625619660' entity='client.admin' 2024-03-10T00:47:12.339 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:47:12 smithi106 ceph-mon[25742]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-10T00:47:13.044 INFO:tasks.cephadm:Distributing conf and client.admin keyring to all hosts + 0755 2024-03-10T00:47:13.044 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:831bc188fa69f143bbba69ac76119e784500e8e3 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 75c37498-de77-11ee-95c7-87774f69a715 -- ceph orch client-keyring set client.admin '*' --mode 0755 2024-03-10T00:47:13.339 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:47:13 smithi106 ceph-mon[25742]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:47:13.645 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/75c37498-de77-11ee-95c7-87774f69a715/mon.a/config 2024-03-10T00:47:14.836 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:47:14 smithi106 ceph-mon[25742]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:47:14.836 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:47:14 smithi106 ceph-mon[25742]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:47:17.589 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:47:17 smithi106 ceph-mon[25742]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:47:17.589 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:47:17 smithi106 ceph-mon[25742]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:47:17.589 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:47:17 smithi106 ceph-mon[25742]: from='client.14176 -' entity='client.admin' cmd=[{"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}]: dispatch 2024-03-10T00:47:17.589 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:47:17 smithi106 ceph-mon[25742]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:47:17.589 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:47:17 smithi106 ceph-mon[25742]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-10T00:47:17.589 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:47:17 smithi106 ceph-mon[25742]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-10T00:47:17.589 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:47:17 smithi106 ceph-mon[25742]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-10T00:47:17.717 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi170 2024-03-10T00:47:17.718 DEBUG:teuthology.orchestra.run.smithi170:> set -ex 2024-03-10T00:47:17.718 DEBUG:teuthology.orchestra.run.smithi170:> dd of=/etc/ceph/ceph.conf 2024-03-10T00:47:17.736 DEBUG:teuthology.orchestra.run.smithi170:> set -ex 2024-03-10T00:47:17.736 DEBUG:teuthology.orchestra.run.smithi170:> dd of=/etc/ceph/ceph.client.admin.keyring 2024-03-10T00:47:17.794 INFO:tasks.cephadm:Adding host smithi170 to orchestrator... 2024-03-10T00:47:17.794 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:831bc188fa69f143bbba69ac76119e784500e8e3 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 75c37498-de77-11ee-95c7-87774f69a715 -- ceph orch host add smithi170 2024-03-10T00:47:18.029 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/75c37498-de77-11ee-95c7-87774f69a715/mon.a/config 2024-03-10T00:47:19.089 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:47:18 smithi106 ceph-mon[25742]: Updating smithi106:/etc/ceph/ceph.conf 2024-03-10T00:47:19.089 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:47:18 smithi106 ceph-mon[25742]: Updating smithi106:/var/lib/ceph/75c37498-de77-11ee-95c7-87774f69a715/config/ceph.conf 2024-03-10T00:47:19.089 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:47:18 smithi106 ceph-mon[25742]: Updating smithi106:/etc/ceph/ceph.client.admin.keyring 2024-03-10T00:47:19.089 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:47:18 smithi106 ceph-mon[25742]: Updating smithi106:/var/lib/ceph/75c37498-de77-11ee-95c7-87774f69a715/config/ceph.client.admin.keyring 2024-03-10T00:47:19.089 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:47:18 smithi106 ceph-mon[25742]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:47:19.089 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:47:18 smithi106 ceph-mon[25742]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:47:19.089 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:47:18 smithi106 ceph-mon[25742]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:47:20.339 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:47:19 smithi106 ceph-mon[25742]: from='client.14178 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi170", "target": ["mon-mgr", ""]}]: dispatch 2024-03-10T00:47:22.089 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:47:21 smithi106 ceph-mon[25742]: Deploying cephadm binary to smithi170 2024-03-10T00:47:22.089 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:47:21 smithi106 ceph-mon[25742]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-10T00:47:24.078 INFO:teuthology.orchestra.run.smithi106.stdout:Added host 'smithi170' with addr '172.21.15.170' 2024-03-10T00:47:24.079 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:47:23 smithi106 ceph-mon[25742]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-10T00:47:24.677 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:831bc188fa69f143bbba69ac76119e784500e8e3 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 75c37498-de77-11ee-95c7-87774f69a715 -- ceph orch host ls --format=json 2024-03-10T00:47:24.913 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/75c37498-de77-11ee-95c7-87774f69a715/mon.a/config 2024-03-10T00:47:25.288 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:47:25 smithi106 ceph-mon[25742]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:47:25.288 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:47:25 smithi106 ceph-mon[25742]: Added host smithi170 2024-03-10T00:47:25.288 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:47:25 smithi106 ceph-mon[25742]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-10T00:47:25.288 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:47:25 smithi106 ceph-mon[25742]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:47:25.288 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:47:25 smithi106 ceph-mon[25742]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-10T00:47:25.289 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:47:25 smithi106 ceph-mon[25742]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:47:26.302 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:47:26 smithi106 ceph-mon[25742]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:47:26.302 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:47:26 smithi106 ceph-mon[25742]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi170", "name": "osd_memory_target"}]: dispatch 2024-03-10T00:47:26.302 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:47:26 smithi106 ceph-mon[25742]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-10T00:47:26.302 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:47:26 smithi106 ceph-mon[25742]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-10T00:47:26.302 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:47:26 smithi106 ceph-mon[25742]: Updating smithi170:/etc/ceph/ceph.conf 2024-03-10T00:47:26.302 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:47:26 smithi106 ceph-mon[25742]: Updating smithi170:/var/lib/ceph/75c37498-de77-11ee-95c7-87774f69a715/config/ceph.conf 2024-03-10T00:47:26.302 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:47:26 smithi106 ceph-mon[25742]: Updating smithi170:/etc/ceph/ceph.client.admin.keyring 2024-03-10T00:47:26.302 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:47:26 smithi106 ceph-mon[25742]: Updating smithi170:/var/lib/ceph/75c37498-de77-11ee-95c7-87774f69a715/config/ceph.client.admin.keyring 2024-03-10T00:47:26.303 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:47:26 smithi106 ceph-mon[25742]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:47:26.303 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:47:26 smithi106 ceph-mon[25742]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:47:26.303 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:47:26 smithi106 ceph-mon[25742]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.agent.smithi170", "caps": []}]: dispatch 2024-03-10T00:47:26.303 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:47:26 smithi106 ceph-mon[25742]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.agent.smithi170", "caps": []}]': finished 2024-03-10T00:47:26.739 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-03-10T00:47:26.740 INFO:teuthology.orchestra.run.smithi106.stdout:[{"addr": "172.21.15.106", "hostname": "smithi106", "labels": [], "status": ""}, {"addr": "172.21.15.170", "hostname": "smithi170", "labels": [], "status": ""}] 2024-03-10T00:47:27.339 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:47:27 smithi106 ceph-mon[25742]: Deploying daemon agent.smithi170 on smithi170 2024-03-10T00:47:27.339 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:47:27 smithi106 ceph-mon[25742]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-10T00:47:27.339 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:47:27 smithi106 ceph-mon[25742]: from='client.14180 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-10T00:47:27.339 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:47:27 smithi106 ceph-mon[25742]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:47:27.339 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:47:27 smithi106 ceph-mon[25742]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:47:27.339 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:47:27 smithi106 ceph-mon[25742]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:47:27.402 INFO:tasks.cephadm:Setting crush tunables to default 2024-03-10T00:47:27.402 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:831bc188fa69f143bbba69ac76119e784500e8e3 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 75c37498-de77-11ee-95c7-87774f69a715 -- ceph osd crush tunables default 2024-03-10T00:47:27.634 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/75c37498-de77-11ee-95c7-87774f69a715/mon.a/config 2024-03-10T00:47:28.263 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:47:28 smithi106 ceph-mon[25742]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-10T00:47:28.263 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:47:28 smithi106 ceph-mon[25742]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-10T00:47:28.263 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:47:28 smithi106 ceph-mon[25742]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-10T00:47:28.263 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:47:28 smithi106 ceph-mon[25742]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:47:28.264 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:47:28 smithi106 ceph-mon[25742]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-10T00:47:28.264 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:47:28 smithi106 ceph-mon[25742]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-10T00:47:28.264 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:47:28 smithi106 ceph-mon[25742]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-10T00:47:28.264 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:47:28 smithi106 ceph-mon[25742]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:47:28.264 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:47:28 smithi106 ceph-mon[25742]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:47:28.264 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:47:28 smithi106 ceph-mon[25742]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:47:28.264 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:47:28 smithi106 ceph-mon[25742]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:47:29.339 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:47:29 smithi106 ceph-mon[25742]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-10T00:47:30.097 INFO:teuthology.orchestra.run.smithi106.stderr:adjusted tunables profile to default 2024-03-10T00:47:30.399 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:47:30 smithi106 ceph-mon[25742]: from='client.? 172.21.15.106:0/3394949640' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-03-10T00:47:30.670 INFO:tasks.cephadm:Adding mon.a on smithi106 2024-03-10T00:47:30.670 INFO:tasks.cephadm:Adding mon.c on smithi106 2024-03-10T00:47:30.670 INFO:tasks.cephadm:Adding mon.b on smithi170 2024-03-10T00:47:30.671 DEBUG:teuthology.orchestra.run.smithi170:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:831bc188fa69f143bbba69ac76119e784500e8e3 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 75c37498-de77-11ee-95c7-87774f69a715 -- ceph orch apply mon '3;smithi106:172.21.15.106=a;smithi106:[v2:172.21.15.106:3301,v1:172.21.15.106:6790]=c;smithi170:172.21.15.170=b' 2024-03-10T00:47:30.899 INFO:teuthology.orchestra.run.smithi170.stderr:Inferring config /var/lib/ceph/75c37498-de77-11ee-95c7-87774f69a715/config/ceph.conf 2024-03-10T00:47:31.428 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:47:31 smithi106 ceph-mon[25742]: from='client.? 172.21.15.106:0/3394949640' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-03-10T00:47:31.428 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:47:31 smithi106 ceph-mon[25742]: osdmap e4: 0 total, 0 up, 0 in 2024-03-10T00:47:31.428 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:47:31 smithi106 ceph-mon[25742]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:47:31.428 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:47:31 smithi106 ceph-mon[25742]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:47:31.428 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:47:31 smithi106 ceph-mon[25742]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-10T00:47:31.428 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:47:31 smithi106 ceph-mon[25742]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:47:32.457 INFO:teuthology.orchestra.run.smithi170.stdout:Scheduled mon update... 2024-03-10T00:47:33.028 DEBUG:teuthology.orchestra.run.smithi106:mon.c> sudo journalctl -f -n 0 -u ceph-75c37498-de77-11ee-95c7-87774f69a715@mon.c.service 2024-03-10T00:47:33.030 DEBUG:teuthology.orchestra.run.smithi170:mon.b> sudo journalctl -f -n 0 -u ceph-75c37498-de77-11ee-95c7-87774f69a715@mon.b.service 2024-03-10T00:47:33.033 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-03-10T00:47:33.033 DEBUG:teuthology.orchestra.run.smithi170:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:831bc188fa69f143bbba69ac76119e784500e8e3 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 75c37498-de77-11ee-95c7-87774f69a715 -- ceph mon dump -f json 2024-03-10T00:47:33.339 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:47:33 smithi106 ceph-mon[25742]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:47:33.339 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:47:33 smithi106 ceph-mon[25742]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:47:33.339 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:47:33 smithi106 ceph-mon[25742]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:47:33.339 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:47:33 smithi106 ceph-mon[25742]: from='client.14184 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi106:172.21.15.106=a;smithi106:[v2:172.21.15.106:3301,v1:172.21.15.106:6790]=c;smithi170:172.21.15.170=b", "target": ["mon-mgr", ""]}]: dispatch 2024-03-10T00:47:33.339 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:47:33 smithi106 ceph-mon[25742]: Saving service mon spec with placement smithi106:172.21.15.106=a;smithi106:[v2:172.21.15.106:3301,v1:172.21.15.106:6790]=c;smithi170:172.21.15.170=b;count:3 2024-03-10T00:47:33.339 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:47:33 smithi106 ceph-mon[25742]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:47:33.339 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:47:33 smithi106 ceph-mon[25742]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-10T00:47:33.340 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:47:33 smithi106 ceph-mon[25742]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-10T00:47:33.340 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:47:33 smithi106 ceph-mon[25742]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-10T00:47:33.340 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:47:33 smithi106 ceph-mon[25742]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:47:33.340 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:47:33 smithi106 ceph-mon[25742]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-10T00:47:33.340 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:47:33 smithi106 ceph-mon[25742]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-10T00:47:33.340 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:47:33 smithi106 ceph-mon[25742]: Deploying daemon mon.b on smithi170 2024-03-10T00:47:33.340 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:47:33 smithi106 ceph-mon[25742]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-10T00:47:33.717 INFO:teuthology.orchestra.run.smithi170.stderr:Inferring config /var/lib/ceph/75c37498-de77-11ee-95c7-87774f69a715/config/ceph.conf 2024-03-10T00:47:36.089 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:47:35 smithi106 ceph-mon[25742]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-10T00:47:36.807 INFO:teuthology.orchestra.run.smithi170.stdout: 2024-03-10T00:47:36.807 INFO:teuthology.orchestra.run.smithi170.stdout:{"epoch":1,"fsid":"75c37498-de77-11ee-95c7-87774f69a715","modified":"2024-03-10T00:45:55.067658Z","created":"2024-03-10T00:45:55.067658Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:3300","nonce":0},{"type":"v1","addr":"172.21.15.106:6789","nonce":0}]},"addr":"172.21.15.106:6789/0","public_addr":"172.21.15.106:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-03-10T00:47:36.809 INFO:teuthology.orchestra.run.smithi170.stderr:dumped monmap epoch 1 2024-03-10T00:47:37.525 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:47:37 smithi170 systemd[1]: Starting Ceph mon.b for 75c37498-de77-11ee-95c7-87774f69a715... 2024-03-10T00:47:37.967 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:47:37 smithi170 podman[27582]: 2024-03-10 00:47:37.75954858 +0000 UTC m=+0.018197353 image pull 01a4f6a6bb3e92ab9ab39a2f835684ea9b10e6ec9fb81ac08ebea4165583a538 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bc4a24690ce5d7255d490e21058d8c114b55e1b93a3d8345f90b6700ad29dada 2024-03-10T00:47:37.967 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:47:37 smithi170 podman[27582]: 2024-03-10 00:47:37.862690005 +0000 UTC m=+0.121338811 container create 4883b5f0e4191157e18a4724a34785eeba6ba990ee71d8e3351825484ed9079d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bc4a24690ce5d7255d490e21058d8c114b55e1b93a3d8345f90b6700ad29dada, name=ceph-75c37498-de77-11ee-95c7-87774f69a715-mon-b, org.label-schema.build-date=20240222, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, ceph=True, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.license=GPLv2, RELEASE=wip-yuri4-testing-2024-03-05-0854-831bc18, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0) 2024-03-10T00:47:38.089 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:47:37 smithi106 ceph-mon[25742]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-10T00:47:38.089 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:47:37 smithi106 ceph-mon[25742]: from='client.? 172.21.15.170:0/1931631842' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-10T00:47:38.525 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:47:38 smithi170 podman[27582]: 2024-03-10 00:47:38.056620839 +0000 UTC m=+0.315269606 container init 4883b5f0e4191157e18a4724a34785eeba6ba990ee71d8e3351825484ed9079d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bc4a24690ce5d7255d490e21058d8c114b55e1b93a3d8345f90b6700ad29dada, name=ceph-75c37498-de77-11ee-95c7-87774f69a715-mon-b, org.label-schema.license=GPLv2, org.label-schema.build-date=20240222, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, GIT_CLEAN=True, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-yuri4-testing-2024-03-05-0854-831bc18, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, ceph=True, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS) 2024-03-10T00:47:38.525 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:47:38 smithi170 podman[27582]: 2024-03-10 00:47:38.062160347 +0000 UTC m=+0.320809113 container start 4883b5f0e4191157e18a4724a34785eeba6ba990ee71d8e3351825484ed9079d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bc4a24690ce5d7255d490e21058d8c114b55e1b93a3d8345f90b6700ad29dada, name=ceph-75c37498-de77-11ee-95c7-87774f69a715-mon-b, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.license=GPLv2, RELEASE=wip-yuri4-testing-2024-03-05-0854-831bc18, org.label-schema.build-date=20240222, maintainer=Guillaume Abrioux ) 2024-03-10T00:47:38.525 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:47:38 smithi170 ceph-mon[27620]: set uid:gid to 167:167 (ceph:ceph) 2024-03-10T00:47:38.526 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:47:38 smithi170 ceph-mon[27620]: ceph version 19.0.0-1875-g831bc188 (831bc188fa69f143bbba69ac76119e784500e8e3) squid (dev), process ceph-mon, pid 7 2024-03-10T00:47:38.526 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:47:38 smithi170 ceph-mon[27620]: pidfile_write: ignore empty --pid-file 2024-03-10T00:47:38.526 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:47:38 smithi170 ceph-mon[27620]: load: jerasure load: lrc 2024-03-10T00:47:38.526 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:47:38 smithi170 ceph-mon[27620]: rocksdb: RocksDB version: 7.9.2 2024-03-10T00:47:38.526 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:47:38 smithi170 ceph-mon[27620]: rocksdb: Git sha 0 2024-03-10T00:47:38.526 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:47:38 smithi170 ceph-mon[27620]: rocksdb: Compile date 2024-03-06 01:10:31 2024-03-10T00:47:38.526 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:47:38 smithi170 ceph-mon[27620]: rocksdb: DB SUMMARY 2024-03-10T00:47:38.526 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:47:38 smithi170 ceph-mon[27620]: rocksdb: DB Session ID: XA1J2Y8U60YBM99GVQ25 2024-03-10T00:47:38.526 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:47:38 smithi170 ceph-mon[27620]: rocksdb: CURRENT file: CURRENT 2024-03-10T00:47:38.526 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:47:38 smithi170 ceph-mon[27620]: rocksdb: IDENTITY file: IDENTITY 2024-03-10T00:47:38.526 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:47:38 smithi170 ceph-mon[27620]: rocksdb: MANIFEST file: MANIFEST-000005 size: 59 Bytes 2024-03-10T00:47:38.526 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:47:38 smithi170 ceph-mon[27620]: rocksdb: SST files in /var/lib/ceph/mon/ceph-b/store.db dir, Total Num: 0, files: 2024-03-10T00:47:38.527 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:47:38 smithi170 ceph-mon[27620]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-b/store.db: 000004.log size: 511 ; 2024-03-10T00:47:38.527 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:47:38 smithi170 ceph-mon[27620]: rocksdb: Options.error_if_exists: 0 2024-03-10T00:47:38.527 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:47:38 smithi170 ceph-mon[27620]: rocksdb: Options.create_if_missing: 0 2024-03-10T00:47:38.527 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:47:38 smithi170 ceph-mon[27620]: rocksdb: Options.paranoid_checks: 1 2024-03-10T00:47:38.527 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:47:38 smithi170 ceph-mon[27620]: rocksdb: Options.flush_verify_memtable_count: 1 2024-03-10T00:47:38.527 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:47:38 smithi170 ceph-mon[27620]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-03-10T00:47:38.527 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:47:38 smithi170 ceph-mon[27620]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-03-10T00:47:38.527 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:47:38 smithi170 ceph-mon[27620]: rocksdb: Options.env: 0x55624a353b20 2024-03-10T00:47:38.527 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:47:38 smithi170 ceph-mon[27620]: rocksdb: Options.fs: PosixFileSystem 2024-03-10T00:47:38.527 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:47:38 smithi170 ceph-mon[27620]: rocksdb: Options.info_log: 0x55624af178e0 2024-03-10T00:47:38.527 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:47:38 smithi170 ceph-mon[27620]: rocksdb: Options.max_file_opening_threads: 16 2024-03-10T00:47:38.529 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:47:38 smithi170 ceph-mon[27620]: rocksdb: Options.statistics: (nil) 2024-03-10T00:47:38.529 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:47:38 smithi170 ceph-mon[27620]: rocksdb: Options.use_fsync: 0 2024-03-10T00:47:38.529 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:47:38 smithi170 ceph-mon[27620]: rocksdb: Options.max_log_file_size: 0 2024-03-10T00:47:38.529 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:47:38 smithi170 ceph-mon[27620]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-03-10T00:47:38.529 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:47:38 smithi170 ceph-mon[27620]: rocksdb: Options.log_file_time_to_roll: 0 2024-03-10T00:47:38.529 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:47:38 smithi170 ceph-mon[27620]: rocksdb: Options.keep_log_file_num: 1000 2024-03-10T00:47:38.529 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:47:38 smithi170 ceph-mon[27620]: rocksdb: Options.recycle_log_file_num: 0 2024-03-10T00:47:38.529 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:47:38 smithi170 ceph-mon[27620]: rocksdb: Options.allow_fallocate: 1 2024-03-10T00:47:38.529 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:47:38 smithi170 ceph-mon[27620]: rocksdb: Options.allow_mmap_reads: 0 2024-03-10T00:47:38.529 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:47:38 smithi170 ceph-mon[27620]: rocksdb: Options.allow_mmap_writes: 0 2024-03-10T00:47:38.529 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:47:38 smithi170 ceph-mon[27620]: rocksdb: Options.use_direct_reads: 0 2024-03-10T00:47:38.530 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:47:38 smithi170 ceph-mon[27620]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-03-10T00:47:38.530 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:47:38 smithi170 ceph-mon[27620]: rocksdb: Options.create_missing_column_families: 0 2024-03-10T00:47:38.530 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:47:38 smithi170 ceph-mon[27620]: rocksdb: Options.db_log_dir: 2024-03-10T00:47:38.530 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:47:38 smithi170 ceph-mon[27620]: rocksdb: Options.wal_dir: 2024-03-10T00:47:38.530 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:47:38 smithi170 ceph-mon[27620]: rocksdb: Options.table_cache_numshardbits: 6 2024-03-10T00:47:38.530 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:47:38 smithi170 ceph-mon[27620]: rocksdb: Options.WAL_ttl_seconds: 0 2024-03-10T00:47:38.530 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:47:38 smithi170 ceph-mon[27620]: rocksdb: Options.WAL_size_limit_MB: 0 2024-03-10T00:47:38.530 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:47:38 smithi170 ceph-mon[27620]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-03-10T00:47:38.530 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:47:38 smithi170 ceph-mon[27620]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-03-10T00:47:38.530 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:47:38 smithi170 ceph-mon[27620]: rocksdb: Options.is_fd_close_on_exec: 1 2024-03-10T00:47:38.530 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:47:38 smithi170 ceph-mon[27620]: rocksdb: Options.advise_random_on_open: 1 2024-03-10T00:47:38.530 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:47:38 smithi170 ceph-mon[27620]: rocksdb: Options.db_write_buffer_size: 0 2024-03-10T00:47:38.530 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:47:38 smithi170 ceph-mon[27620]: rocksdb: Options.write_buffer_manager: 0x55624aeeb400 2024-03-10T00:47:38.530 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:47:38 smithi170 ceph-mon[27620]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-03-10T00:47:38.531 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:47:38 smithi170 ceph-mon[27620]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-03-10T00:47:38.531 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:47:38 smithi170 ceph-mon[27620]: rocksdb: Options.use_adaptive_mutex: 0 2024-03-10T00:47:38.531 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:47:38 smithi170 ceph-mon[27620]: rocksdb: Options.rate_limiter: (nil) 2024-03-10T00:47:38.531 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:47:38 smithi170 ceph-mon[27620]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-03-10T00:47:38.531 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:47:38 smithi170 ceph-mon[27620]: rocksdb: Options.wal_recovery_mode: 2 2024-03-10T00:47:38.531 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:47:38 smithi170 ceph-mon[27620]: rocksdb: Options.enable_thread_tracking: 0 2024-03-10T00:47:38.531 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:47:38 smithi170 ceph-mon[27620]: rocksdb: Options.enable_pipelined_write: 0 2024-03-10T00:47:38.531 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:47:38 smithi170 ceph-mon[27620]: rocksdb: Options.unordered_write: 0 2024-03-10T00:47:38.531 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:47:38 smithi170 ceph-mon[27620]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-03-10T00:47:38.531 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:47:38 smithi170 ceph-mon[27620]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-03-10T00:47:38.531 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:47:38 smithi170 ceph-mon[27620]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-03-10T00:47:38.531 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:47:38 smithi170 ceph-mon[27620]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-03-10T00:47:38.531 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:47:38 smithi170 ceph-mon[27620]: rocksdb: Options.row_cache: None 2024-03-10T00:47:38.531 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:47:38 smithi170 ceph-mon[27620]: rocksdb: Options.wal_filter: None 2024-03-10T00:47:38.532 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:47:38 smithi170 ceph-mon[27620]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-03-10T00:47:38.532 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:47:38 smithi170 ceph-mon[27620]: rocksdb: Options.allow_ingest_behind: 0 2024-03-10T00:47:38.532 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:47:38 smithi170 ceph-mon[27620]: rocksdb: Options.two_write_queues: 0 2024-03-10T00:47:38.532 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:47:38 smithi170 ceph-mon[27620]: rocksdb: Options.manual_wal_flush: 0 2024-03-10T00:47:38.532 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:47:38 smithi170 ceph-mon[27620]: rocksdb: Options.wal_compression: 0 2024-03-10T00:47:38.532 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:47:38 smithi170 ceph-mon[27620]: rocksdb: Options.atomic_flush: 0 2024-03-10T00:47:38.532 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:47:38 smithi170 ceph-mon[27620]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-03-10T00:47:38.532 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:47:38 smithi170 ceph-mon[27620]: rocksdb: Options.persist_stats_to_disk: 0 2024-03-10T00:47:38.532 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:47:38 smithi170 ceph-mon[27620]: rocksdb: Options.write_dbid_to_manifest: 0 2024-03-10T00:47:38.533 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:47:38 smithi170 ceph-mon[27620]: rocksdb: Options.log_readahead_size: 0 2024-03-10T00:47:38.533 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:47:38 smithi170 ceph-mon[27620]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-03-10T00:47:38.533 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:47:38 smithi170 ceph-mon[27620]: rocksdb: Options.best_efforts_recovery: 0 2024-03-10T00:47:38.533 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:47:38 smithi170 ceph-mon[27620]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-03-10T00:47:38.533 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:47:38 smithi170 ceph-mon[27620]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-03-10T00:47:38.533 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:47:38 smithi170 ceph-mon[27620]: rocksdb: Options.allow_data_in_errors: 0 2024-03-10T00:47:38.533 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:47:38 smithi170 ceph-mon[27620]: rocksdb: Options.db_host_id: __hostname__ 2024-03-10T00:47:38.533 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:47:38 smithi170 ceph-mon[27620]: rocksdb: Options.enforce_single_del_contracts: true 2024-03-10T00:47:38.533 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:47:38 smithi170 ceph-mon[27620]: rocksdb: Options.max_background_jobs: 2 2024-03-10T00:47:38.533 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:47:38 smithi170 ceph-mon[27620]: rocksdb: Options.max_background_compactions: -1 2024-03-10T00:47:38.533 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:47:38 smithi170 ceph-mon[27620]: rocksdb: Options.max_subcompactions: 1 2024-03-10T00:47:38.533 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:47:38 smithi170 ceph-mon[27620]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-03-10T00:47:38.533 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:47:38 smithi170 ceph-mon[27620]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-03-10T00:47:38.533 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:47:38 smithi170 ceph-mon[27620]: rocksdb: Options.delayed_write_rate : 16777216 2024-03-10T00:47:38.533 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:47:38 smithi170 ceph-mon[27620]: rocksdb: Options.max_total_wal_size: 0 2024-03-10T00:47:38.534 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:47:38 smithi170 ceph-mon[27620]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-03-10T00:47:38.534 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:47:38 smithi170 ceph-mon[27620]: rocksdb: Options.stats_dump_period_sec: 600 2024-03-10T00:47:38.534 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:47:38 smithi170 ceph-mon[27620]: rocksdb: Options.stats_persist_period_sec: 600 2024-03-10T00:47:38.534 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:47:38 smithi170 ceph-mon[27620]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-03-10T00:47:38.534 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:47:38 smithi170 ceph-mon[27620]: rocksdb: Options.max_open_files: -1 2024-03-10T00:47:38.534 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:47:38 smithi170 ceph-mon[27620]: rocksdb: Options.bytes_per_sync: 0 2024-03-10T00:47:38.534 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:47:38 smithi170 ceph-mon[27620]: rocksdb: Options.wal_bytes_per_sync: 0 2024-03-10T00:47:38.534 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:47:38 smithi170 ceph-mon[27620]: rocksdb: Options.strict_bytes_per_sync: 0 2024-03-10T00:47:38.534 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:47:38 smithi170 ceph-mon[27620]: rocksdb: Options.compaction_readahead_size: 0 2024-03-10T00:47:38.534 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:47:38 smithi170 ceph-mon[27620]: rocksdb: Options.max_background_flushes: -1 2024-03-10T00:47:38.534 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:47:38 smithi170 ceph-mon[27620]: rocksdb: Compression algorithms supported: 2024-03-10T00:47:38.534 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:47:38 smithi170 ceph-mon[27620]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-03-10T00:47:38.534 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:47:38 smithi170 ceph-mon[27620]: rocksdb: kZSTD supported: 0 2024-03-10T00:47:38.535 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:47:38 smithi170 ceph-mon[27620]: rocksdb: kXpressCompression supported: 0 2024-03-10T00:47:38.535 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:47:38 smithi170 ceph-mon[27620]: rocksdb: kLZ4HCCompression supported: 1 2024-03-10T00:47:38.535 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:47:38 smithi170 ceph-mon[27620]: rocksdb: kZlibCompression supported: 1 2024-03-10T00:47:38.535 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:47:38 smithi170 ceph-mon[27620]: rocksdb: kSnappyCompression supported: 1 2024-03-10T00:47:38.535 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:47:38 smithi170 ceph-mon[27620]: rocksdb: kLZ4Compression supported: 1 2024-03-10T00:47:38.535 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:47:38 smithi170 ceph-mon[27620]: rocksdb: kBZip2Compression supported: 0 2024-03-10T00:47:38.535 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:47:38 smithi170 ceph-mon[27620]: rocksdb: Fast CRC32 supported: Supported on x86 2024-03-10T00:47:38.535 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:47:38 smithi170 ceph-mon[27620]: rocksdb: DMutex implementation: pthread_mutex_t 2024-03-10T00:47:38.535 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:47:38 smithi170 ceph-mon[27620]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-b/store.db/MANIFEST-000005 2024-03-10T00:47:38.535 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:47:38 smithi170 ceph-mon[27620]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-03-10T00:47:38.535 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:47:38 smithi170 ceph-mon[27620]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-03-10T00:47:38.535 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:47:38 smithi170 ceph-mon[27620]: rocksdb: Options.merge_operator: 2024-03-10T00:47:38.535 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:47:38 smithi170 ceph-mon[27620]: rocksdb: Options.compaction_filter: None 2024-03-10T00:47:38.535 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:47:38 smithi170 ceph-mon[27620]: rocksdb: Options.compaction_filter_factory: None 2024-03-10T00:47:38.536 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:47:38 smithi170 ceph-mon[27620]: rocksdb: Options.sst_partitioner_factory: None 2024-03-10T00:47:38.536 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:47:38 smithi170 ceph-mon[27620]: rocksdb: Options.memtable_factory: SkipListFactory 2024-03-10T00:47:38.536 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:47:38 smithi170 ceph-mon[27620]: rocksdb: Options.table_factory: BlockBasedTable 2024-03-10T00:47:38.536 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:47:38 smithi170 ceph-mon[27620]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x55624af179e0) 2024-03-10T00:47:38.536 INFO:journalctl@ceph.mon.b.smithi170.stdout: cache_index_and_filter_blocks: 1 2024-03-10T00:47:38.536 INFO:journalctl@ceph.mon.b.smithi170.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-03-10T00:47:38.536 INFO:journalctl@ceph.mon.b.smithi170.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-03-10T00:47:38.536 INFO:journalctl@ceph.mon.b.smithi170.stdout: pin_top_level_index_and_filter: 1 2024-03-10T00:47:38.536 INFO:journalctl@ceph.mon.b.smithi170.stdout: index_type: 0 2024-03-10T00:47:38.536 INFO:journalctl@ceph.mon.b.smithi170.stdout: data_block_index_type: 0 2024-03-10T00:47:38.536 INFO:journalctl@ceph.mon.b.smithi170.stdout: index_shortening: 1 2024-03-10T00:47:38.536 INFO:journalctl@ceph.mon.b.smithi170.stdout: data_block_hash_table_util_ratio: 0.750000 2024-03-10T00:47:38.536 INFO:journalctl@ceph.mon.b.smithi170.stdout: checksum: 4 2024-03-10T00:47:38.536 INFO:journalctl@ceph.mon.b.smithi170.stdout: no_block_cache: 0 2024-03-10T00:47:38.537 INFO:journalctl@ceph.mon.b.smithi170.stdout: block_cache: 0x55624af1d090 2024-03-10T00:47:38.537 INFO:journalctl@ceph.mon.b.smithi170.stdout: block_cache_name: BinnedLRUCache 2024-03-10T00:47:38.537 INFO:journalctl@ceph.mon.b.smithi170.stdout: block_cache_options: 2024-03-10T00:47:38.537 INFO:journalctl@ceph.mon.b.smithi170.stdout: capacity : 536870912 2024-03-10T00:47:38.537 INFO:journalctl@ceph.mon.b.smithi170.stdout: num_shard_bits : 4 2024-03-10T00:47:38.537 INFO:journalctl@ceph.mon.b.smithi170.stdout: strict_capacity_limit : 0 2024-03-10T00:47:38.537 INFO:journalctl@ceph.mon.b.smithi170.stdout: high_pri_pool_ratio: 0.000 2024-03-10T00:47:38.537 INFO:journalctl@ceph.mon.b.smithi170.stdout: block_cache_compressed: (nil) 2024-03-10T00:47:38.537 INFO:journalctl@ceph.mon.b.smithi170.stdout: persistent_cache: (nil) 2024-03-10T00:47:38.537 INFO:journalctl@ceph.mon.b.smithi170.stdout: block_size: 4096 2024-03-10T00:47:38.537 INFO:journalctl@ceph.mon.b.smithi170.stdout: block_size_deviation: 10 2024-03-10T00:47:38.537 INFO:journalctl@ceph.mon.b.smithi170.stdout: block_restart_interval: 16 2024-03-10T00:47:38.537 INFO:journalctl@ceph.mon.b.smithi170.stdout: index_block_restart_interval: 1 2024-03-10T00:47:38.537 INFO:journalctl@ceph.mon.b.smithi170.stdout: metadata_block_size: 4096 2024-03-10T00:47:38.538 INFO:journalctl@ceph.mon.b.smithi170.stdout: partition_filters: 0 2024-03-10T00:47:38.538 INFO:journalctl@ceph.mon.b.smithi170.stdout: use_delta_encoding: 1 2024-03-10T00:47:38.538 INFO:journalctl@ceph.mon.b.smithi170.stdout: filter_policy: bloomfilter 2024-03-10T00:47:38.538 INFO:journalctl@ceph.mon.b.smithi170.stdout: whole_key_filtering: 1 2024-03-10T00:47:38.538 INFO:journalctl@ceph.mon.b.smithi170.stdout: verify_compression: 0 2024-03-10T00:47:38.538 INFO:journalctl@ceph.mon.b.smithi170.stdout: read_amp_bytes_per_bit: 0 2024-03-10T00:47:38.538 INFO:journalctl@ceph.mon.b.smithi170.stdout: format_version: 5 2024-03-10T00:47:38.538 INFO:journalctl@ceph.mon.b.smithi170.stdout: enable_index_compression: 1 2024-03-10T00:47:38.538 INFO:journalctl@ceph.mon.b.smithi170.stdout: block_align: 0 2024-03-10T00:47:38.538 INFO:journalctl@ceph.mon.b.smithi170.stdout: max_auto_readahead_size: 262144 2024-03-10T00:47:38.538 INFO:journalctl@ceph.mon.b.smithi170.stdout: prepopulate_block_cache: 0 2024-03-10T00:47:38.538 INFO:journalctl@ceph.mon.b.smithi170.stdout: initial_auto_readahead_size: 8192 2024-03-10T00:47:38.538 INFO:journalctl@ceph.mon.b.smithi170.stdout: num_file_reads_for_auto_readahead: 2 2024-03-10T00:47:38.538 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:47:38 smithi170 ceph-mon[27620]: rocksdb: Options.write_buffer_size: 33554432 2024-03-10T00:47:38.538 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:47:38 smithi170 ceph-mon[27620]: rocksdb: Options.max_write_buffer_number: 2 2024-03-10T00:47:38.539 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:47:38 smithi170 ceph-mon[27620]: rocksdb: Options.compression: NoCompression 2024-03-10T00:47:38.539 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:47:38 smithi170 ceph-mon[27620]: rocksdb: Options.bottommost_compression: Disabled 2024-03-10T00:47:38.539 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:47:38 smithi170 ceph-mon[27620]: rocksdb: Options.prefix_extractor: nullptr 2024-03-10T00:47:38.539 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:47:38 smithi170 ceph-mon[27620]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-03-10T00:47:38.539 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:47:38 smithi170 ceph-mon[27620]: rocksdb: Options.num_levels: 7 2024-03-10T00:47:38.539 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:47:38 smithi170 ceph-mon[27620]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-03-10T00:47:38.539 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:47:38 smithi170 ceph-mon[27620]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-03-10T00:47:38.539 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:47:38 smithi170 ceph-mon[27620]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-03-10T00:47:38.539 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:47:38 smithi170 ceph-mon[27620]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-03-10T00:47:38.539 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:47:38 smithi170 ceph-mon[27620]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-03-10T00:47:38.539 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:47:38 smithi170 ceph-mon[27620]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-03-10T00:47:38.539 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:47:38 smithi170 ceph-mon[27620]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-03-10T00:47:38.539 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:47:38 smithi170 ceph-mon[27620]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-03-10T00:47:38.539 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:47:38 smithi170 ceph-mon[27620]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-03-10T00:47:38.540 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:47:38 smithi170 ceph-mon[27620]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-03-10T00:47:38.540 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:47:38 smithi170 ceph-mon[27620]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-03-10T00:47:38.540 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:47:38 smithi170 ceph-mon[27620]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-03-10T00:47:38.540 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:47:38 smithi170 ceph-mon[27620]: rocksdb: Options.compression_opts.window_bits: -14 2024-03-10T00:47:38.540 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:47:38 smithi170 ceph-mon[27620]: rocksdb: Options.compression_opts.level: 32767 2024-03-10T00:47:38.540 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:47:38 smithi170 ceph-mon[27620]: rocksdb: Options.compression_opts.strategy: 0 2024-03-10T00:47:38.540 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:47:38 smithi170 ceph-mon[27620]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-03-10T00:47:38.540 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:47:38 smithi170 ceph-mon[27620]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-03-10T00:47:38.540 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:47:38 smithi170 ceph-mon[27620]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-03-10T00:47:38.540 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:47:38 smithi170 ceph-mon[27620]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-03-10T00:47:38.540 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:47:38 smithi170 ceph-mon[27620]: rocksdb: Options.compression_opts.enabled: false 2024-03-10T00:47:38.540 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:47:38 smithi170 ceph-mon[27620]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-03-10T00:47:38.540 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:47:38 smithi170 ceph-mon[27620]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-03-10T00:47:38.540 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:47:38 smithi170 ceph-mon[27620]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-03-10T00:47:38.540 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:47:38 smithi170 ceph-mon[27620]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-03-10T00:47:38.541 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:47:38 smithi170 ceph-mon[27620]: rocksdb: Options.target_file_size_base: 67108864 2024-03-10T00:47:38.541 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:47:38 smithi170 ceph-mon[27620]: rocksdb: Options.target_file_size_multiplier: 1 2024-03-10T00:47:38.541 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:47:38 smithi170 ceph-mon[27620]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-03-10T00:47:38.541 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:47:38 smithi170 ceph-mon[27620]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-03-10T00:47:38.541 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:47:38 smithi170 ceph-mon[27620]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-03-10T00:47:38.541 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:47:38 smithi170 ceph-mon[27620]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-03-10T00:47:38.541 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:47:38 smithi170 ceph-mon[27620]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-03-10T00:47:38.541 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:47:38 smithi170 ceph-mon[27620]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-03-10T00:47:38.541 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:47:38 smithi170 ceph-mon[27620]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-03-10T00:47:38.541 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:47:38 smithi170 ceph-mon[27620]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-03-10T00:47:38.541 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:47:38 smithi170 ceph-mon[27620]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-03-10T00:47:38.541 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:47:38 smithi170 ceph-mon[27620]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-03-10T00:47:38.541 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:47:38 smithi170 ceph-mon[27620]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-03-10T00:47:38.542 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:47:38 smithi170 ceph-mon[27620]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-03-10T00:47:38.542 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:47:38 smithi170 ceph-mon[27620]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-03-10T00:47:38.542 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:47:38 smithi170 ceph-mon[27620]: rocksdb: Options.arena_block_size: 1048576 2024-03-10T00:47:38.542 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:47:38 smithi170 ceph-mon[27620]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-03-10T00:47:38.542 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:47:38 smithi170 ceph-mon[27620]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-03-10T00:47:38.542 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:47:38 smithi170 ceph-mon[27620]: rocksdb: Options.disable_auto_compactions: 0 2024-03-10T00:47:38.542 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:47:38 smithi170 ceph-mon[27620]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-03-10T00:47:38.542 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:47:38 smithi170 ceph-mon[27620]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-03-10T00:47:38.542 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:47:38 smithi170 ceph-mon[27620]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-03-10T00:47:38.542 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:47:38 smithi170 ceph-mon[27620]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-03-10T00:47:38.542 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:47:38 smithi170 ceph-mon[27620]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-03-10T00:47:38.542 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:47:38 smithi170 ceph-mon[27620]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-03-10T00:47:38.542 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:47:38 smithi170 ceph-mon[27620]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-03-10T00:47:38.543 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:47:38 smithi170 ceph-mon[27620]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-03-10T00:47:38.543 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:47:38 smithi170 ceph-mon[27620]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-03-10T00:47:38.543 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:47:38 smithi170 ceph-mon[27620]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-03-10T00:47:38.543 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:47:38 smithi170 ceph-mon[27620]: rocksdb: Options.table_properties_collectors: 2024-03-10T00:47:38.543 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:47:38 smithi170 ceph-mon[27620]: rocksdb: Options.inplace_update_support: 0 2024-03-10T00:47:38.543 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:47:38 smithi170 ceph-mon[27620]: rocksdb: Options.inplace_update_num_locks: 10000 2024-03-10T00:47:38.543 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:47:38 smithi170 ceph-mon[27620]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-03-10T00:47:38.543 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:47:38 smithi170 ceph-mon[27620]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-03-10T00:47:38.543 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:47:38 smithi170 ceph-mon[27620]: rocksdb: Options.memtable_huge_page_size: 0 2024-03-10T00:47:38.543 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:47:38 smithi170 ceph-mon[27620]: rocksdb: Options.bloom_locality: 0 2024-03-10T00:47:38.543 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:47:38 smithi170 ceph-mon[27620]: rocksdb: Options.max_successive_merges: 0 2024-03-10T00:47:38.543 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:47:38 smithi170 ceph-mon[27620]: rocksdb: Options.optimize_filters_for_hits: 0 2024-03-10T00:47:38.543 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:47:38 smithi170 ceph-mon[27620]: rocksdb: Options.paranoid_file_checks: 0 2024-03-10T00:47:38.543 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:47:38 smithi170 ceph-mon[27620]: rocksdb: Options.force_consistency_checks: 1 2024-03-10T00:47:38.543 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:47:38 smithi170 ceph-mon[27620]: rocksdb: Options.report_bg_io_stats: 0 2024-03-10T00:47:38.544 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:47:38 smithi170 ceph-mon[27620]: rocksdb: Options.ttl: 2592000 2024-03-10T00:47:38.544 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:47:38 smithi170 ceph-mon[27620]: rocksdb: Options.periodic_compaction_seconds: 0 2024-03-10T00:47:38.544 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:47:38 smithi170 ceph-mon[27620]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-03-10T00:47:38.544 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:47:38 smithi170 ceph-mon[27620]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-03-10T00:47:38.544 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:47:38 smithi170 ceph-mon[27620]: rocksdb: Options.enable_blob_files: false 2024-03-10T00:47:38.544 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:47:38 smithi170 ceph-mon[27620]: rocksdb: Options.min_blob_size: 0 2024-03-10T00:47:38.544 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:47:38 smithi170 ceph-mon[27620]: rocksdb: Options.blob_file_size: 268435456 2024-03-10T00:47:38.544 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:47:38 smithi170 ceph-mon[27620]: rocksdb: Options.blob_compression_type: NoCompression 2024-03-10T00:47:38.544 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:47:38 smithi170 ceph-mon[27620]: rocksdb: Options.enable_blob_garbage_collection: false 2024-03-10T00:47:38.544 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:47:38 smithi170 ceph-mon[27620]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-03-10T00:47:38.544 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:47:38 smithi170 ceph-mon[27620]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-03-10T00:47:38.544 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:47:38 smithi170 ceph-mon[27620]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-03-10T00:47:38.544 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:47:38 smithi170 ceph-mon[27620]: rocksdb: Options.blob_file_starting_level: 0 2024-03-10T00:47:38.544 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:47:38 smithi170 ceph-mon[27620]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-03-10T00:47:38.544 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:47:38 smithi170 ceph-mon[27620]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-b/store.db/MANIFEST-000005 succeeded,manifest_file_number is 5, next_file_number is 7, last_sequence is 0, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-03-10T00:47:38.545 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:47:38 smithi170 ceph-mon[27620]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 0 2024-03-10T00:47:38.545 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:47:38 smithi170 ceph-mon[27620]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: 6da6b63f-8ca9-4600-89b8-7ca9ead36cd4 2024-03-10T00:47:38.545 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:47:38 smithi170 ceph-mon[27620]: rocksdb: EVENT_LOG_v1 {"time_micros": 1710031658092258, "job": 1, "event": "recovery_started", "wal_files": [4]} 2024-03-10T00:47:38.545 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:47:38 smithi170 ceph-mon[27620]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #4 mode 2 2024-03-10T00:47:38.545 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:47:38 smithi170 ceph-mon[27620]: rocksdb: EVENT_LOG_v1 {"time_micros": 1710031658092666, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 8, "file_size": 1622, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 1, "largest_seqno": 5, "table_properties": {"data_size": 523, "index_size": 31, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 69, "raw_key_size": 115, "raw_average_key_size": 23, "raw_value_size": 401, "raw_average_value_size": 80, "num_data_blocks": 1, "num_entries": 5, "num_filter_entries": 5, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1710031658, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "6da6b63f-8ca9-4600-89b8-7ca9ead36cd4", "db_session_id": "XA1J2Y8U60YBM99GVQ25", "orig_file_number": 8, "seqno_to_time_mapping": "N/A"}} 2024-03-10T00:47:38.545 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:47:38 smithi170 ceph-mon[27620]: rocksdb: EVENT_LOG_v1 {"time_micros": 1710031658092745, "job": 1, "event": "recovery_finished"} 2024-03-10T00:47:38.545 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:47:38 smithi170 ceph-mon[27620]: rocksdb: [db/version_set.cc:5047] Creating manifest 10 2024-03-10T00:47:38.545 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:47:38 smithi170 ceph-mon[27620]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-b/store.db/000004.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-03-10T00:47:38.545 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:47:38 smithi170 ceph-mon[27620]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x55624b010000 2024-03-10T00:47:38.545 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:47:38 smithi170 ceph-mon[27620]: rocksdb: DB pointer 0x55624affa000 2024-03-10T00:47:38.545 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:47:38 smithi170 ceph-mon[27620]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-03-10T00:47:38.545 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:47:38 smithi170 ceph-mon[27620]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-03-10T00:47:38.545 INFO:journalctl@ceph.mon.b.smithi170.stdout: ** DB Stats ** 2024-03-10T00:47:38.545 INFO:journalctl@ceph.mon.b.smithi170.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-03-10T00:47:38.546 INFO:journalctl@ceph.mon.b.smithi170.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-03-10T00:47:38.546 INFO:journalctl@ceph.mon.b.smithi170.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-03-10T00:47:38.546 INFO:journalctl@ceph.mon.b.smithi170.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-03-10T00:47:38.546 INFO:journalctl@ceph.mon.b.smithi170.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-03-10T00:47:38.546 INFO:journalctl@ceph.mon.b.smithi170.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-03-10T00:47:38.546 INFO:journalctl@ceph.mon.b.smithi170.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-03-10T00:47:38.546 INFO:journalctl@ceph.mon.b.smithi170.stdout: 2024-03-10T00:47:38.546 INFO:journalctl@ceph.mon.b.smithi170.stdout: ** Compaction Stats [default] ** 2024-03-10T00:47:38.546 INFO:journalctl@ceph.mon.b.smithi170.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-03-10T00:47:38.546 INFO:journalctl@ceph.mon.b.smithi170.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-03-10T00:47:38.546 INFO:journalctl@ceph.mon.b.smithi170.stdout: L0 1/0 1.58 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 4.0 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-03-10T00:47:38.546 INFO:journalctl@ceph.mon.b.smithi170.stdout: Sum 1/0 1.58 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 4.0 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-03-10T00:47:38.546 INFO:journalctl@ceph.mon.b.smithi170.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 4.0 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-03-10T00:47:38.546 INFO:journalctl@ceph.mon.b.smithi170.stdout: 2024-03-10T00:47:38.547 INFO:journalctl@ceph.mon.b.smithi170.stdout: ** Compaction Stats [default] ** 2024-03-10T00:47:38.547 INFO:journalctl@ceph.mon.b.smithi170.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-03-10T00:47:38.547 INFO:journalctl@ceph.mon.b.smithi170.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-03-10T00:47:38.547 INFO:journalctl@ceph.mon.b.smithi170.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 4.0 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-03-10T00:47:38.547 INFO:journalctl@ceph.mon.b.smithi170.stdout: 2024-03-10T00:47:38.547 INFO:journalctl@ceph.mon.b.smithi170.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-03-10T00:47:38.547 INFO:journalctl@ceph.mon.b.smithi170.stdout: 2024-03-10T00:47:38.547 INFO:journalctl@ceph.mon.b.smithi170.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-03-10T00:47:38.547 INFO:journalctl@ceph.mon.b.smithi170.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-03-10T00:47:38.547 INFO:journalctl@ceph.mon.b.smithi170.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-03-10T00:47:38.547 INFO:journalctl@ceph.mon.b.smithi170.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-03-10T00:47:38.547 INFO:journalctl@ceph.mon.b.smithi170.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-03-10T00:47:38.547 INFO:journalctl@ceph.mon.b.smithi170.stdout: AddFile(Keys): cumulative 0, interval 0 2024-03-10T00:47:38.547 INFO:journalctl@ceph.mon.b.smithi170.stdout: Cumulative compaction: 0.00 GB write, 0.64 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-10T00:47:38.547 INFO:journalctl@ceph.mon.b.smithi170.stdout: Interval compaction: 0.00 GB write, 0.64 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-10T00:47:38.548 INFO:journalctl@ceph.mon.b.smithi170.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-03-10T00:47:38.548 INFO:journalctl@ceph.mon.b.smithi170.stdout: Block cache BinnedLRUCache@0x55624af1d090#7 capacity: 512.00 MB usage: 0.22 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 9e-06 secs_since: 0 2024-03-10T00:47:38.548 INFO:journalctl@ceph.mon.b.smithi170.stdout: Block cache entry stats(count,size,portion): FilterBlock(1,0.11 KB,2.08616e-05%) IndexBlock(1,0.11 KB,2.08616e-05%) Misc(1,0.00 KB,0%) 2024-03-10T00:47:38.548 INFO:journalctl@ceph.mon.b.smithi170.stdout: 2024-03-10T00:47:38.548 INFO:journalctl@ceph.mon.b.smithi170.stdout: ** File Read Latency Histogram By Level [default] ** 2024-03-10T00:47:38.548 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:47:38 smithi170 ceph-mon[27620]: mon.b does not exist in monmap, will attempt to join an existing cluster 2024-03-10T00:47:38.548 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:47:38 smithi170 ceph-mon[27620]: using public_addr v2:172.21.15.170:0/0 -> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] 2024-03-10T00:47:38.548 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:47:38 smithi170 ceph-mon[27620]: starting mon.b rank -1 at public addrs [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] at bind addrs [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] mon_data /var/lib/ceph/mon/ceph-b fsid 75c37498-de77-11ee-95c7-87774f69a715 2024-03-10T00:47:38.548 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:47:38 smithi170 ceph-mon[27620]: mon.b@-1(???) e0 preinit fsid 75c37498-de77-11ee-95c7-87774f69a715 2024-03-10T00:47:38.548 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:47:38 smithi170 ceph-mon[27620]: mon.b@-1(synchronizing).mds e1 new map 2024-03-10T00:47:38.548 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:47:38 smithi170 ceph-mon[27620]: mon.b@-1(synchronizing).mds e1 print_map 2024-03-10T00:47:38.548 INFO:journalctl@ceph.mon.b.smithi170.stdout: e1 2024-03-10T00:47:38.548 INFO:journalctl@ceph.mon.b.smithi170.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-03-10T00:47:38.548 INFO:journalctl@ceph.mon.b.smithi170.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments} 2024-03-10T00:47:38.549 INFO:journalctl@ceph.mon.b.smithi170.stdout: legacy client fscid: -1 2024-03-10T00:47:38.549 INFO:journalctl@ceph.mon.b.smithi170.stdout: 2024-03-10T00:47:38.549 INFO:journalctl@ceph.mon.b.smithi170.stdout: No filesystems configured 2024-03-10T00:47:38.549 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:47:38 smithi170 ceph-mon[27620]: mon.b@-1(synchronizing).osd e0 _set_cache_ratios kv ratio 0.25 inc ratio 0.375 full ratio 0.375 2024-03-10T00:47:38.549 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:47:38 smithi170 ceph-mon[27620]: mon.b@-1(synchronizing).osd e0 register_cache_with_pcm pcm target: 2147483648 pcm max: 1020054732 pcm min: 134217728 inc_osd_cache size: 1 2024-03-10T00:47:38.549 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:47:38 smithi170 ceph-mon[27620]: mon.b@-1(synchronizing).osd e1 e1: 0 total, 0 up, 0 in 2024-03-10T00:47:38.549 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:47:38 smithi170 ceph-mon[27620]: mon.b@-1(synchronizing).osd e2 e2: 0 total, 0 up, 0 in 2024-03-10T00:47:38.549 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:47:38 smithi170 ceph-mon[27620]: mon.b@-1(synchronizing).osd e3 e3: 0 total, 0 up, 0 in 2024-03-10T00:47:38.549 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:47:38 smithi170 ceph-mon[27620]: mon.b@-1(synchronizing).osd e4 e4: 0 total, 0 up, 0 in 2024-03-10T00:47:38.549 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:47:38 smithi170 ceph-mon[27620]: mon.b@-1(synchronizing).osd e4 crush map has features 3314932999778484224, adjusting msgr requires 2024-03-10T00:47:38.549 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:47:38 smithi170 ceph-mon[27620]: mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-03-10T00:47:38.549 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:47:38 smithi170 ceph-mon[27620]: mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-03-10T00:47:38.549 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:47:38 smithi170 ceph-mon[27620]: mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-03-10T00:47:38.549 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:47:38 smithi170 ceph-mon[27620]: from='client.14178 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi170", "target": ["mon-mgr", ""]}]: dispatch 2024-03-10T00:47:38.549 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:47:38 smithi170 ceph-mon[27620]: Deploying cephadm binary to smithi170 2024-03-10T00:47:38.550 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:47:38 smithi170 ceph-mon[27620]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-10T00:47:38.550 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:47:38 smithi170 ceph-mon[27620]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-10T00:47:38.550 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:47:38 smithi170 ceph-mon[27620]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:47:38.550 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:47:38 smithi170 ceph-mon[27620]: Added host smithi170 2024-03-10T00:47:38.550 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:47:38 smithi170 ceph-mon[27620]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-10T00:47:38.550 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:47:38 smithi170 ceph-mon[27620]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:47:38.550 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:47:38 smithi170 ceph-mon[27620]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-10T00:47:38.550 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:47:38 smithi170 ceph-mon[27620]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:47:38.550 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:47:38 smithi170 ceph-mon[27620]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:47:38.550 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:47:38 smithi170 ceph-mon[27620]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi170", "name": "osd_memory_target"}]: dispatch 2024-03-10T00:47:38.550 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:47:38 smithi170 ceph-mon[27620]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-10T00:47:38.550 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:47:38 smithi170 ceph-mon[27620]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-10T00:47:38.550 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:47:38 smithi170 ceph-mon[27620]: Updating smithi170:/etc/ceph/ceph.conf 2024-03-10T00:47:38.550 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:47:38 smithi170 ceph-mon[27620]: Updating smithi170:/var/lib/ceph/75c37498-de77-11ee-95c7-87774f69a715/config/ceph.conf 2024-03-10T00:47:38.551 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:47:38 smithi170 ceph-mon[27620]: Updating smithi170:/etc/ceph/ceph.client.admin.keyring 2024-03-10T00:47:38.551 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:47:38 smithi170 ceph-mon[27620]: Updating smithi170:/var/lib/ceph/75c37498-de77-11ee-95c7-87774f69a715/config/ceph.client.admin.keyring 2024-03-10T00:47:38.551 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:47:38 smithi170 ceph-mon[27620]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:47:38.551 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:47:38 smithi170 ceph-mon[27620]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:47:38.551 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:47:38 smithi170 ceph-mon[27620]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.agent.smithi170", "caps": []}]: dispatch 2024-03-10T00:47:38.551 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:47:38 smithi170 ceph-mon[27620]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.agent.smithi170", "caps": []}]': finished 2024-03-10T00:47:38.551 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:47:38 smithi170 ceph-mon[27620]: Deploying daemon agent.smithi170 on smithi170 2024-03-10T00:47:38.551 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:47:38 smithi170 ceph-mon[27620]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-10T00:47:38.551 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:47:38 smithi170 ceph-mon[27620]: from='client.14180 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-10T00:47:38.551 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:47:38 smithi170 ceph-mon[27620]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:47:38.551 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:47:38 smithi170 ceph-mon[27620]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:47:38.551 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:47:38 smithi170 ceph-mon[27620]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:47:38.551 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:47:38 smithi170 ceph-mon[27620]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-10T00:47:38.551 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:47:38 smithi170 ceph-mon[27620]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-10T00:47:38.552 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:47:38 smithi170 ceph-mon[27620]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-10T00:47:38.552 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:47:38 smithi170 ceph-mon[27620]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:47:38.552 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:47:38 smithi170 ceph-mon[27620]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-10T00:47:38.552 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:47:38 smithi170 ceph-mon[27620]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-10T00:47:38.552 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:47:38 smithi170 ceph-mon[27620]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-10T00:47:38.553 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:47:38 smithi170 ceph-mon[27620]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:47:38.553 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:47:38 smithi170 ceph-mon[27620]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:47:38.553 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:47:38 smithi170 ceph-mon[27620]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:47:38.553 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:47:38 smithi170 ceph-mon[27620]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:47:38.553 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:47:38 smithi170 ceph-mon[27620]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-10T00:47:38.553 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:47:38 smithi170 ceph-mon[27620]: from='client.? 172.21.15.106:0/3394949640' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-03-10T00:47:38.553 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:47:38 smithi170 ceph-mon[27620]: from='client.? 172.21.15.106:0/3394949640' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-03-10T00:47:38.553 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:47:38 smithi170 ceph-mon[27620]: osdmap e4: 0 total, 0 up, 0 in 2024-03-10T00:47:38.553 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:47:38 smithi170 ceph-mon[27620]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:47:38.554 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:47:38 smithi170 ceph-mon[27620]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:47:38.554 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:47:38 smithi170 ceph-mon[27620]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-10T00:47:38.554 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:47:38 smithi170 ceph-mon[27620]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:47:38.554 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:47:38 smithi170 ceph-mon[27620]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:47:38.554 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:47:38 smithi170 ceph-mon[27620]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:47:38.554 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:47:38 smithi170 ceph-mon[27620]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:47:38.554 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:47:38 smithi170 ceph-mon[27620]: from='client.14184 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi106:172.21.15.106=a;smithi106:[v2:172.21.15.106:3301,v1:172.21.15.106:6790]=c;smithi170:172.21.15.170=b", "target": ["mon-mgr", ""]}]: dispatch 2024-03-10T00:47:38.554 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:47:38 smithi170 ceph-mon[27620]: Saving service mon spec with placement smithi106:172.21.15.106=a;smithi106:[v2:172.21.15.106:3301,v1:172.21.15.106:6790]=c;smithi170:172.21.15.170=b;count:3 2024-03-10T00:47:38.554 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:47:38 smithi170 ceph-mon[27620]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:47:38.554 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:47:38 smithi170 ceph-mon[27620]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-10T00:47:38.554 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:47:38 smithi170 ceph-mon[27620]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-10T00:47:38.554 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:47:38 smithi170 ceph-mon[27620]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-10T00:47:38.554 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:47:38 smithi170 ceph-mon[27620]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:47:38.554 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:47:38 smithi170 ceph-mon[27620]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-10T00:47:38.554 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:47:38 smithi170 ceph-mon[27620]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-10T00:47:38.555 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:47:38 smithi170 ceph-mon[27620]: Deploying daemon mon.b on smithi170 2024-03-10T00:47:38.555 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:47:38 smithi170 ceph-mon[27620]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-10T00:47:38.555 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:47:38 smithi170 ceph-mon[27620]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-10T00:47:38.555 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:47:38 smithi170 ceph-mon[27620]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-10T00:47:38.555 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:47:38 smithi170 ceph-mon[27620]: from='client.? 172.21.15.170:0/1931631842' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-10T00:47:38.555 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:47:38 smithi170 ceph-mon[27620]: mon.b@-1(synchronizing).paxosservice(auth 1..5) refresh upgraded, format 0 -> 3 2024-03-10T00:47:38.555 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:47:38 smithi170 bash[27582]: 4883b5f0e4191157e18a4724a34785eeba6ba990ee71d8e3351825484ed9079d 2024-03-10T00:47:38.555 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:47:38 smithi170 systemd[1]: Started Ceph mon.b for 75c37498-de77-11ee-95c7-87774f69a715. 2024-03-10T00:47:38.653 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-03-10T00:47:38.653 DEBUG:teuthology.orchestra.run.smithi170:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:831bc188fa69f143bbba69ac76119e784500e8e3 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 75c37498-de77-11ee-95c7-87774f69a715 -- ceph mon dump -f json 2024-03-10T00:47:38.889 INFO:teuthology.orchestra.run.smithi170.stderr:Inferring config /var/lib/ceph/75c37498-de77-11ee-95c7-87774f69a715/mon.b/config 2024-03-10T00:47:43.525 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:47:43 smithi170 ceph-mon[27620]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-10T00:47:43.525 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:47:43 smithi170 ceph-mon[27620]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-10T00:47:43.525 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:47:43 smithi170 ceph-mon[27620]: mon.a calling monitor election 2024-03-10T00:47:43.525 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:47:43 smithi170 ceph-mon[27620]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-10T00:47:43.525 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:47:43 smithi170 ceph-mon[27620]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-10T00:47:43.526 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:47:43 smithi170 ceph-mon[27620]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-10T00:47:43.526 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:47:43 smithi170 ceph-mon[27620]: mon.b calling monitor election 2024-03-10T00:47:43.526 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:47:43 smithi170 ceph-mon[27620]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-10T00:47:43.526 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:47:43 smithi170 ceph-mon[27620]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-10T00:47:43.526 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:47:43 smithi170 ceph-mon[27620]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-10T00:47:43.526 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:47:43 smithi170 ceph-mon[27620]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-10T00:47:43.526 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:47:43 smithi170 ceph-mon[27620]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-10T00:47:43.526 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:47:43 smithi170 ceph-mon[27620]: mon.a is new leader, mons a,b in quorum (ranks 0,1) 2024-03-10T00:47:43.526 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:47:43 smithi170 ceph-mon[27620]: monmap e2: 2 mons at {a=[v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0],b=[v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-10T00:47:43.526 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:47:43 smithi170 ceph-mon[27620]: fsmap 2024-03-10T00:47:43.526 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:47:43 smithi170 ceph-mon[27620]: osdmap e4: 0 total, 0 up, 0 in 2024-03-10T00:47:43.526 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:47:43 smithi170 ceph-mon[27620]: mgrmap e13: y(active, since 42s) 2024-03-10T00:47:43.526 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:47:43 smithi170 ceph-mon[27620]: overall HEALTH_OK 2024-03-10T00:47:43.526 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:47:43 smithi170 ceph-mon[27620]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:47:43.527 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:47:43 smithi170 ceph-mon[27620]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:47:43.527 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:47:43 smithi170 ceph-mon[27620]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:47:43.527 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:47:43 smithi170 ceph-mon[27620]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-10T00:47:43.527 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:47:43 smithi170 ceph-mon[27620]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-10T00:47:43.589 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:47:43 smithi106 ceph-mon[25742]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-10T00:47:43.589 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:47:43 smithi106 ceph-mon[25742]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-10T00:47:43.589 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:47:43 smithi106 ceph-mon[25742]: mon.a calling monitor election 2024-03-10T00:47:43.589 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:47:43 smithi106 ceph-mon[25742]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-10T00:47:43.589 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:47:43 smithi106 ceph-mon[25742]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-10T00:47:43.589 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:47:43 smithi106 ceph-mon[25742]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-10T00:47:43.589 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:47:43 smithi106 ceph-mon[25742]: mon.b calling monitor election 2024-03-10T00:47:43.590 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:47:43 smithi106 ceph-mon[25742]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-10T00:47:43.590 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:47:43 smithi106 ceph-mon[25742]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-10T00:47:43.590 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:47:43 smithi106 ceph-mon[25742]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-10T00:47:43.590 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:47:43 smithi106 ceph-mon[25742]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-10T00:47:43.590 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:47:43 smithi106 ceph-mon[25742]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-10T00:47:43.590 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:47:43 smithi106 ceph-mon[25742]: mon.a is new leader, mons a,b in quorum (ranks 0,1) 2024-03-10T00:47:43.590 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:47:43 smithi106 ceph-mon[25742]: monmap e2: 2 mons at {a=[v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0],b=[v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-10T00:47:43.590 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:47:43 smithi106 ceph-mon[25742]: fsmap 2024-03-10T00:47:43.590 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:47:43 smithi106 ceph-mon[25742]: osdmap e4: 0 total, 0 up, 0 in 2024-03-10T00:47:43.590 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:47:43 smithi106 ceph-mon[25742]: mgrmap e13: y(active, since 42s) 2024-03-10T00:47:43.590 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:47:43 smithi106 ceph-mon[25742]: overall HEALTH_OK 2024-03-10T00:47:43.590 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:47:43 smithi106 ceph-mon[25742]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:47:43.590 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:47:43 smithi106 ceph-mon[25742]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:47:43.591 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:47:43 smithi106 ceph-mon[25742]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:47:43.591 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:47:43 smithi106 ceph-mon[25742]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-10T00:47:43.591 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:47:43 smithi106 ceph-mon[25742]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-10T00:47:43.591 INFO:teuthology.orchestra.run.smithi170.stdout: 2024-03-10T00:47:43.591 INFO:teuthology.orchestra.run.smithi170.stdout:{"epoch":2,"fsid":"75c37498-de77-11ee-95c7-87774f69a715","modified":"2024-03-10T00:47:38.116666Z","created":"2024-03-10T00:45:55.067658Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:3300","nonce":0},{"type":"v1","addr":"172.21.15.106:6789","nonce":0}]},"addr":"172.21.15.106:6789/0","public_addr":"172.21.15.106:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:3300","nonce":0},{"type":"v1","addr":"172.21.15.170:6789","nonce":0}]},"addr":"172.21.15.170:6789/0","public_addr":"172.21.15.170:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2024-03-10T00:47:43.592 INFO:teuthology.orchestra.run.smithi170.stderr:dumped monmap epoch 2 2024-03-10T00:47:44.472 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:47:44 smithi106 ceph-mon[25742]: Deploying daemon mon.c on smithi106 2024-03-10T00:47:44.473 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:47:44 smithi106 ceph-mon[25742]: from='client.? 172.21.15.170:0/919666768' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-10T00:47:44.473 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:47:44 smithi106 ceph-mon[25742]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-10T00:47:44.524 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:47:44 smithi170 ceph-mon[27620]: Deploying daemon mon.c on smithi106 2024-03-10T00:47:44.525 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:47:44 smithi170 ceph-mon[27620]: from='client.? 172.21.15.170:0/919666768' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-10T00:47:44.525 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:47:44 smithi170 ceph-mon[27620]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-10T00:47:45.144 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-03-10T00:47:45.145 DEBUG:teuthology.orchestra.run.smithi170:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:831bc188fa69f143bbba69ac76119e784500e8e3 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 75c37498-de77-11ee-95c7-87774f69a715 -- ceph mon dump -f json 2024-03-10T00:47:45.339 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:47:45 smithi106 ceph-mon[25742]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-10T00:47:45.387 INFO:teuthology.orchestra.run.smithi170.stderr:Inferring config /var/lib/ceph/75c37498-de77-11ee-95c7-87774f69a715/mon.b/config 2024-03-10T00:47:45.508 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:47:45 smithi170 ceph-mon[27620]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-10T00:47:46.339 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:47:46 smithi106 systemd[1]: Starting Ceph mon.c for 75c37498-de77-11ee-95c7-87774f69a715... 2024-03-10T00:47:46.697 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:47:46 smithi106 podman[33422]: 2024-03-10 00:47:46.336801879 +0000 UTC m=+0.017836059 image pull 01a4f6a6bb3e92ab9ab39a2f835684ea9b10e6ec9fb81ac08ebea4165583a538 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bc4a24690ce5d7255d490e21058d8c114b55e1b93a3d8345f90b6700ad29dada 2024-03-10T00:47:46.697 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:47:46 smithi106 podman[33422]: 2024-03-10 00:47:46.449216658 +0000 UTC m=+0.130250829 container create 2411e393082d5ae1f677c7a188bf2ee903ed532e4f6db2cd90cc776ace2c1493 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bc4a24690ce5d7255d490e21058d8c114b55e1b93a3d8345f90b6700ad29dada, name=ceph-75c37498-de77-11ee-95c7-87774f69a715-mon-c, ceph=True, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, RELEASE=wip-yuri4-testing-2024-03-05-0854-831bc18, org.label-schema.build-date=20240222, org.label-schema.license=GPLv2) 2024-03-10T00:47:46.697 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:47:46 smithi106 podman[33422]: 2024-03-10 00:47:46.66824719 +0000 UTC m=+0.349281373 container init 2411e393082d5ae1f677c7a188bf2ee903ed532e4f6db2cd90cc776ace2c1493 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bc4a24690ce5d7255d490e21058d8c114b55e1b93a3d8345f90b6700ad29dada, name=ceph-75c37498-de77-11ee-95c7-87774f69a715-mon-c, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, RELEASE=wip-yuri4-testing-2024-03-05-0854-831bc18, org.label-schema.build-date=20240222, ceph=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux ) 2024-03-10T00:47:46.697 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:47:46 smithi106 podman[33422]: 2024-03-10 00:47:46.67420592 +0000 UTC m=+0.355240095 container start 2411e393082d5ae1f677c7a188bf2ee903ed532e4f6db2cd90cc776ace2c1493 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bc4a24690ce5d7255d490e21058d8c114b55e1b93a3d8345f90b6700ad29dada, name=ceph-75c37498-de77-11ee-95c7-87774f69a715-mon-c, GIT_CLEAN=True, io.buildah.version=1.33.5, RELEASE=wip-yuri4-testing-2024-03-05-0854-831bc18, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240222, org.label-schema.license=GPLv2) 2024-03-10T00:47:47.089 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:47:46 smithi106 ceph-mon[33460]: set uid:gid to 167:167 (ceph:ceph) 2024-03-10T00:47:47.090 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:47:46 smithi106 ceph-mon[33460]: ceph version 19.0.0-1875-g831bc188 (831bc188fa69f143bbba69ac76119e784500e8e3) squid (dev), process ceph-mon, pid 7 2024-03-10T00:47:47.090 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:47:46 smithi106 ceph-mon[33460]: pidfile_write: ignore empty --pid-file 2024-03-10T00:47:47.090 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:47:46 smithi106 ceph-mon[33460]: load: jerasure load: lrc 2024-03-10T00:47:47.090 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:47:46 smithi106 ceph-mon[33460]: rocksdb: RocksDB version: 7.9.2 2024-03-10T00:47:47.090 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:47:46 smithi106 ceph-mon[33460]: rocksdb: Git sha 0 2024-03-10T00:47:47.090 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:47:46 smithi106 ceph-mon[33460]: rocksdb: Compile date 2024-03-06 01:10:31 2024-03-10T00:47:47.090 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:47:46 smithi106 ceph-mon[33460]: rocksdb: DB SUMMARY 2024-03-10T00:47:47.090 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:47:46 smithi106 ceph-mon[33460]: rocksdb: DB Session ID: V6LMNOAYDPNP337Q668A 2024-03-10T00:47:47.090 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:47:46 smithi106 ceph-mon[33460]: rocksdb: CURRENT file: CURRENT 2024-03-10T00:47:47.090 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:47:46 smithi106 ceph-mon[33460]: rocksdb: IDENTITY file: IDENTITY 2024-03-10T00:47:47.090 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:47:46 smithi106 ceph-mon[33460]: rocksdb: MANIFEST file: MANIFEST-000005 size: 59 Bytes 2024-03-10T00:47:47.090 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:47:46 smithi106 ceph-mon[33460]: rocksdb: SST files in /var/lib/ceph/mon/ceph-c/store.db dir, Total Num: 0, files: 2024-03-10T00:47:47.091 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:47:46 smithi106 ceph-mon[33460]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-c/store.db: 000004.log size: 636 ; 2024-03-10T00:47:47.091 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:47:46 smithi106 ceph-mon[33460]: rocksdb: Options.error_if_exists: 0 2024-03-10T00:47:47.091 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:47:46 smithi106 ceph-mon[33460]: rocksdb: Options.create_if_missing: 0 2024-03-10T00:47:47.091 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:47:46 smithi106 ceph-mon[33460]: rocksdb: Options.paranoid_checks: 1 2024-03-10T00:47:47.091 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:47:46 smithi106 ceph-mon[33460]: rocksdb: Options.flush_verify_memtable_count: 1 2024-03-10T00:47:47.091 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:47:46 smithi106 ceph-mon[33460]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-03-10T00:47:47.091 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:47:46 smithi106 ceph-mon[33460]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-03-10T00:47:47.091 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:47:46 smithi106 ceph-mon[33460]: rocksdb: Options.env: 0x55e3dee30b20 2024-03-10T00:47:47.091 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:47:46 smithi106 ceph-mon[33460]: rocksdb: Options.fs: PosixFileSystem 2024-03-10T00:47:47.091 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:47:46 smithi106 ceph-mon[33460]: rocksdb: Options.info_log: 0x55e3e0a8d8e0 2024-03-10T00:47:47.091 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:47:46 smithi106 ceph-mon[33460]: rocksdb: Options.max_file_opening_threads: 16 2024-03-10T00:47:47.091 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:47:46 smithi106 ceph-mon[33460]: rocksdb: Options.statistics: (nil) 2024-03-10T00:47:47.091 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:47:46 smithi106 ceph-mon[33460]: rocksdb: Options.use_fsync: 0 2024-03-10T00:47:47.092 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:47:46 smithi106 ceph-mon[33460]: rocksdb: Options.max_log_file_size: 0 2024-03-10T00:47:47.092 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:47:46 smithi106 ceph-mon[33460]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-03-10T00:47:47.092 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:47:46 smithi106 ceph-mon[33460]: rocksdb: Options.log_file_time_to_roll: 0 2024-03-10T00:47:47.092 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:47:46 smithi106 ceph-mon[33460]: rocksdb: Options.keep_log_file_num: 1000 2024-03-10T00:47:47.092 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:47:46 smithi106 ceph-mon[33460]: rocksdb: Options.recycle_log_file_num: 0 2024-03-10T00:47:47.092 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:47:46 smithi106 ceph-mon[33460]: rocksdb: Options.allow_fallocate: 1 2024-03-10T00:47:47.092 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:47:46 smithi106 ceph-mon[33460]: rocksdb: Options.allow_mmap_reads: 0 2024-03-10T00:47:47.092 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:47:46 smithi106 ceph-mon[33460]: rocksdb: Options.allow_mmap_writes: 0 2024-03-10T00:47:47.092 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:47:46 smithi106 ceph-mon[33460]: rocksdb: Options.use_direct_reads: 0 2024-03-10T00:47:47.092 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:47:46 smithi106 ceph-mon[33460]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-03-10T00:47:47.092 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:47:46 smithi106 ceph-mon[33460]: rocksdb: Options.create_missing_column_families: 0 2024-03-10T00:47:47.092 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:47:46 smithi106 ceph-mon[33460]: rocksdb: Options.db_log_dir: 2024-03-10T00:47:47.092 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:47:46 smithi106 ceph-mon[33460]: rocksdb: Options.wal_dir: 2024-03-10T00:47:47.093 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:47:46 smithi106 ceph-mon[33460]: rocksdb: Options.table_cache_numshardbits: 6 2024-03-10T00:47:47.093 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:47:46 smithi106 ceph-mon[33460]: rocksdb: Options.WAL_ttl_seconds: 0 2024-03-10T00:47:47.093 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:47:46 smithi106 ceph-mon[33460]: rocksdb: Options.WAL_size_limit_MB: 0 2024-03-10T00:47:47.093 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:47:46 smithi106 ceph-mon[33460]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-03-10T00:47:47.093 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:47:46 smithi106 ceph-mon[33460]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-03-10T00:47:47.093 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:47:46 smithi106 ceph-mon[33460]: rocksdb: Options.is_fd_close_on_exec: 1 2024-03-10T00:47:47.093 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:47:46 smithi106 ceph-mon[33460]: rocksdb: Options.advise_random_on_open: 1 2024-03-10T00:47:47.093 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:47:46 smithi106 ceph-mon[33460]: rocksdb: Options.db_write_buffer_size: 0 2024-03-10T00:47:47.093 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:47:46 smithi106 ceph-mon[33460]: rocksdb: Options.write_buffer_manager: 0x55e3e0a61400 2024-03-10T00:47:47.093 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:47:46 smithi106 ceph-mon[33460]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-03-10T00:47:47.093 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:47:46 smithi106 ceph-mon[33460]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-03-10T00:47:47.094 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:47:46 smithi106 ceph-mon[33460]: rocksdb: Options.use_adaptive_mutex: 0 2024-03-10T00:47:47.094 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:47:46 smithi106 ceph-mon[33460]: rocksdb: Options.rate_limiter: (nil) 2024-03-10T00:47:47.095 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:47:46 smithi106 ceph-mon[33460]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-03-10T00:47:47.095 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:47:46 smithi106 ceph-mon[33460]: rocksdb: Options.wal_recovery_mode: 2 2024-03-10T00:47:47.095 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:47:46 smithi106 ceph-mon[33460]: rocksdb: Options.enable_thread_tracking: 0 2024-03-10T00:47:47.095 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:47:46 smithi106 ceph-mon[33460]: rocksdb: Options.enable_pipelined_write: 0 2024-03-10T00:47:47.095 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:47:46 smithi106 ceph-mon[33460]: rocksdb: Options.unordered_write: 0 2024-03-10T00:47:47.095 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:47:46 smithi106 ceph-mon[33460]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-03-10T00:47:47.095 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:47:46 smithi106 ceph-mon[33460]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-03-10T00:47:47.095 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:47:46 smithi106 ceph-mon[33460]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-03-10T00:47:47.095 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:47:46 smithi106 ceph-mon[33460]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-03-10T00:47:47.095 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:47:46 smithi106 ceph-mon[33460]: rocksdb: Options.row_cache: None 2024-03-10T00:47:47.095 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:47:46 smithi106 ceph-mon[33460]: rocksdb: Options.wal_filter: None 2024-03-10T00:47:47.095 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:47:46 smithi106 ceph-mon[33460]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-03-10T00:47:47.095 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:47:46 smithi106 ceph-mon[33460]: rocksdb: Options.allow_ingest_behind: 0 2024-03-10T00:47:47.095 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:47:46 smithi106 ceph-mon[33460]: rocksdb: Options.two_write_queues: 0 2024-03-10T00:47:47.096 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:47:46 smithi106 ceph-mon[33460]: rocksdb: Options.manual_wal_flush: 0 2024-03-10T00:47:47.096 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:47:46 smithi106 ceph-mon[33460]: rocksdb: Options.wal_compression: 0 2024-03-10T00:47:47.096 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:47:46 smithi106 ceph-mon[33460]: rocksdb: Options.atomic_flush: 0 2024-03-10T00:47:47.096 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:47:46 smithi106 ceph-mon[33460]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-03-10T00:47:47.096 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:47:46 smithi106 ceph-mon[33460]: rocksdb: Options.persist_stats_to_disk: 0 2024-03-10T00:47:47.096 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:47:46 smithi106 ceph-mon[33460]: rocksdb: Options.write_dbid_to_manifest: 0 2024-03-10T00:47:47.096 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:47:46 smithi106 ceph-mon[33460]: rocksdb: Options.log_readahead_size: 0 2024-03-10T00:47:47.096 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:47:46 smithi106 ceph-mon[33460]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-03-10T00:47:47.096 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:47:46 smithi106 ceph-mon[33460]: rocksdb: Options.best_efforts_recovery: 0 2024-03-10T00:47:47.096 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:47:46 smithi106 ceph-mon[33460]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-03-10T00:47:47.096 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:47:46 smithi106 ceph-mon[33460]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-03-10T00:47:47.096 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:47:46 smithi106 ceph-mon[33460]: rocksdb: Options.allow_data_in_errors: 0 2024-03-10T00:47:47.096 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:47:46 smithi106 ceph-mon[33460]: rocksdb: Options.db_host_id: __hostname__ 2024-03-10T00:47:47.097 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:47:46 smithi106 ceph-mon[33460]: rocksdb: Options.enforce_single_del_contracts: true 2024-03-10T00:47:47.098 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:47:46 smithi106 ceph-mon[33460]: rocksdb: Options.max_background_jobs: 2 2024-03-10T00:47:47.098 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:47:46 smithi106 ceph-mon[33460]: rocksdb: Options.max_background_compactions: -1 2024-03-10T00:47:47.098 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:47:46 smithi106 ceph-mon[33460]: rocksdb: Options.max_subcompactions: 1 2024-03-10T00:47:47.099 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:47:46 smithi106 ceph-mon[33460]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-03-10T00:47:47.099 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:47:46 smithi106 ceph-mon[33460]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-03-10T00:47:47.099 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:47:46 smithi106 ceph-mon[33460]: rocksdb: Options.delayed_write_rate : 16777216 2024-03-10T00:47:47.099 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:47:46 smithi106 ceph-mon[33460]: rocksdb: Options.max_total_wal_size: 0 2024-03-10T00:47:47.099 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:47:46 smithi106 ceph-mon[33460]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-03-10T00:47:47.099 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:47:46 smithi106 ceph-mon[33460]: rocksdb: Options.stats_dump_period_sec: 600 2024-03-10T00:47:47.099 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:47:46 smithi106 ceph-mon[33460]: rocksdb: Options.stats_persist_period_sec: 600 2024-03-10T00:47:47.099 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:47:46 smithi106 ceph-mon[33460]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-03-10T00:47:47.099 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:47:46 smithi106 ceph-mon[33460]: rocksdb: Options.max_open_files: -1 2024-03-10T00:47:47.099 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:47:46 smithi106 ceph-mon[33460]: rocksdb: Options.bytes_per_sync: 0 2024-03-10T00:47:47.099 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:47:46 smithi106 ceph-mon[33460]: rocksdb: Options.wal_bytes_per_sync: 0 2024-03-10T00:47:47.099 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:47:46 smithi106 ceph-mon[33460]: rocksdb: Options.strict_bytes_per_sync: 0 2024-03-10T00:47:47.099 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:47:46 smithi106 ceph-mon[33460]: rocksdb: Options.compaction_readahead_size: 0 2024-03-10T00:47:47.099 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:47:46 smithi106 ceph-mon[33460]: rocksdb: Options.max_background_flushes: -1 2024-03-10T00:47:47.100 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:47:46 smithi106 ceph-mon[33460]: rocksdb: Compression algorithms supported: 2024-03-10T00:47:47.100 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:47:46 smithi106 ceph-mon[33460]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-03-10T00:47:47.100 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:47:46 smithi106 ceph-mon[33460]: rocksdb: kZSTD supported: 0 2024-03-10T00:47:47.100 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:47:46 smithi106 ceph-mon[33460]: rocksdb: kXpressCompression supported: 0 2024-03-10T00:47:47.100 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:47:46 smithi106 ceph-mon[33460]: rocksdb: kLZ4HCCompression supported: 1 2024-03-10T00:47:47.100 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:47:46 smithi106 ceph-mon[33460]: rocksdb: kZlibCompression supported: 1 2024-03-10T00:47:47.100 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:47:46 smithi106 ceph-mon[33460]: rocksdb: kSnappyCompression supported: 1 2024-03-10T00:47:47.100 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:47:46 smithi106 ceph-mon[33460]: rocksdb: kLZ4Compression supported: 1 2024-03-10T00:47:47.100 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:47:46 smithi106 ceph-mon[33460]: rocksdb: kBZip2Compression supported: 0 2024-03-10T00:47:47.100 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:47:46 smithi106 ceph-mon[33460]: rocksdb: Fast CRC32 supported: Supported on x86 2024-03-10T00:47:47.100 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:47:46 smithi106 ceph-mon[33460]: rocksdb: DMutex implementation: pthread_mutex_t 2024-03-10T00:47:47.100 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:47:46 smithi106 ceph-mon[33460]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-c/store.db/MANIFEST-000005 2024-03-10T00:47:47.100 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:47:46 smithi106 ceph-mon[33460]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-03-10T00:47:47.100 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:47:46 smithi106 ceph-mon[33460]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-03-10T00:47:47.101 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:47:46 smithi106 ceph-mon[33460]: rocksdb: Options.merge_operator: 2024-03-10T00:47:47.101 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:47:46 smithi106 ceph-mon[33460]: rocksdb: Options.compaction_filter: None 2024-03-10T00:47:47.101 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:47:46 smithi106 ceph-mon[33460]: rocksdb: Options.compaction_filter_factory: None 2024-03-10T00:47:47.101 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:47:46 smithi106 ceph-mon[33460]: rocksdb: Options.sst_partitioner_factory: None 2024-03-10T00:47:47.101 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:47:46 smithi106 ceph-mon[33460]: rocksdb: Options.memtable_factory: SkipListFactory 2024-03-10T00:47:47.101 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:47:46 smithi106 ceph-mon[33460]: rocksdb: Options.table_factory: BlockBasedTable 2024-03-10T00:47:47.101 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:47:46 smithi106 ceph-mon[33460]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x55e3e0a8d9e0) 2024-03-10T00:47:47.101 INFO:journalctl@ceph.mon.c.smithi106.stdout: cache_index_and_filter_blocks: 1 2024-03-10T00:47:47.101 INFO:journalctl@ceph.mon.c.smithi106.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-03-10T00:47:47.101 INFO:journalctl@ceph.mon.c.smithi106.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-03-10T00:47:47.101 INFO:journalctl@ceph.mon.c.smithi106.stdout: pin_top_level_index_and_filter: 1 2024-03-10T00:47:47.101 INFO:journalctl@ceph.mon.c.smithi106.stdout: index_type: 0 2024-03-10T00:47:47.101 INFO:journalctl@ceph.mon.c.smithi106.stdout: data_block_index_type: 0 2024-03-10T00:47:47.101 INFO:journalctl@ceph.mon.c.smithi106.stdout: index_shortening: 1 2024-03-10T00:47:47.101 INFO:journalctl@ceph.mon.c.smithi106.stdout: data_block_hash_table_util_ratio: 0.750000 2024-03-10T00:47:47.102 INFO:journalctl@ceph.mon.c.smithi106.stdout: checksum: 4 2024-03-10T00:47:47.102 INFO:journalctl@ceph.mon.c.smithi106.stdout: no_block_cache: 0 2024-03-10T00:47:47.102 INFO:journalctl@ceph.mon.c.smithi106.stdout: block_cache: 0x55e3e0a93090 2024-03-10T00:47:47.102 INFO:journalctl@ceph.mon.c.smithi106.stdout: block_cache_name: BinnedLRUCache 2024-03-10T00:47:47.102 INFO:journalctl@ceph.mon.c.smithi106.stdout: block_cache_options: 2024-03-10T00:47:47.102 INFO:journalctl@ceph.mon.c.smithi106.stdout: capacity : 536870912 2024-03-10T00:47:47.102 INFO:journalctl@ceph.mon.c.smithi106.stdout: num_shard_bits : 4 2024-03-10T00:47:47.102 INFO:journalctl@ceph.mon.c.smithi106.stdout: strict_capacity_limit : 0 2024-03-10T00:47:47.102 INFO:journalctl@ceph.mon.c.smithi106.stdout: high_pri_pool_ratio: 0.000 2024-03-10T00:47:47.102 INFO:journalctl@ceph.mon.c.smithi106.stdout: block_cache_compressed: (nil) 2024-03-10T00:47:47.102 INFO:journalctl@ceph.mon.c.smithi106.stdout: persistent_cache: (nil) 2024-03-10T00:47:47.102 INFO:journalctl@ceph.mon.c.smithi106.stdout: block_size: 4096 2024-03-10T00:47:47.102 INFO:journalctl@ceph.mon.c.smithi106.stdout: block_size_deviation: 10 2024-03-10T00:47:47.102 INFO:journalctl@ceph.mon.c.smithi106.stdout: block_restart_interval: 16 2024-03-10T00:47:47.102 INFO:journalctl@ceph.mon.c.smithi106.stdout: index_block_restart_interval: 1 2024-03-10T00:47:47.103 INFO:journalctl@ceph.mon.c.smithi106.stdout: metadata_block_size: 4096 2024-03-10T00:47:47.103 INFO:journalctl@ceph.mon.c.smithi106.stdout: partition_filters: 0 2024-03-10T00:47:47.103 INFO:journalctl@ceph.mon.c.smithi106.stdout: use_delta_encoding: 1 2024-03-10T00:47:47.103 INFO:journalctl@ceph.mon.c.smithi106.stdout: filter_policy: bloomfilter 2024-03-10T00:47:47.103 INFO:journalctl@ceph.mon.c.smithi106.stdout: whole_key_filtering: 1 2024-03-10T00:47:47.103 INFO:journalctl@ceph.mon.c.smithi106.stdout: verify_compression: 0 2024-03-10T00:47:47.103 INFO:journalctl@ceph.mon.c.smithi106.stdout: read_amp_bytes_per_bit: 0 2024-03-10T00:47:47.103 INFO:journalctl@ceph.mon.c.smithi106.stdout: format_version: 5 2024-03-10T00:47:47.103 INFO:journalctl@ceph.mon.c.smithi106.stdout: enable_index_compression: 1 2024-03-10T00:47:47.103 INFO:journalctl@ceph.mon.c.smithi106.stdout: block_align: 0 2024-03-10T00:47:47.103 INFO:journalctl@ceph.mon.c.smithi106.stdout: max_auto_readahead_size: 262144 2024-03-10T00:47:47.103 INFO:journalctl@ceph.mon.c.smithi106.stdout: prepopulate_block_cache: 0 2024-03-10T00:47:47.103 INFO:journalctl@ceph.mon.c.smithi106.stdout: initial_auto_readahead_size: 8192 2024-03-10T00:47:47.103 INFO:journalctl@ceph.mon.c.smithi106.stdout: num_file_reads_for_auto_readahead: 2 2024-03-10T00:47:47.104 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:47:46 smithi106 ceph-mon[33460]: rocksdb: Options.write_buffer_size: 33554432 2024-03-10T00:47:47.104 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:47:46 smithi106 ceph-mon[33460]: rocksdb: Options.max_write_buffer_number: 2 2024-03-10T00:47:47.104 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:47:46 smithi106 ceph-mon[33460]: rocksdb: Options.compression: NoCompression 2024-03-10T00:47:47.104 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:47:46 smithi106 ceph-mon[33460]: rocksdb: Options.bottommost_compression: Disabled 2024-03-10T00:47:47.104 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:47:46 smithi106 ceph-mon[33460]: rocksdb: Options.prefix_extractor: nullptr 2024-03-10T00:47:47.104 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:47:46 smithi106 ceph-mon[33460]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-03-10T00:47:47.104 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:47:46 smithi106 ceph-mon[33460]: rocksdb: Options.num_levels: 7 2024-03-10T00:47:47.104 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:47:46 smithi106 ceph-mon[33460]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-03-10T00:47:47.104 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:47:46 smithi106 ceph-mon[33460]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-03-10T00:47:47.104 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:47:46 smithi106 ceph-mon[33460]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-03-10T00:47:47.104 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:47:46 smithi106 ceph-mon[33460]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-03-10T00:47:47.104 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:47:46 smithi106 ceph-mon[33460]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-03-10T00:47:47.104 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:47:46 smithi106 ceph-mon[33460]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-03-10T00:47:47.104 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:47:46 smithi106 ceph-mon[33460]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-03-10T00:47:47.105 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:47:46 smithi106 ceph-mon[33460]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-03-10T00:47:47.105 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:47:46 smithi106 ceph-mon[33460]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-03-10T00:47:47.105 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:47:46 smithi106 ceph-mon[33460]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-03-10T00:47:47.105 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:47:46 smithi106 ceph-mon[33460]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-03-10T00:47:47.105 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:47:46 smithi106 ceph-mon[33460]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-03-10T00:47:47.105 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:47:46 smithi106 ceph-mon[33460]: rocksdb: Options.compression_opts.window_bits: -14 2024-03-10T00:47:47.105 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:47:46 smithi106 ceph-mon[33460]: rocksdb: Options.compression_opts.level: 32767 2024-03-10T00:47:47.105 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:47:46 smithi106 ceph-mon[33460]: rocksdb: Options.compression_opts.strategy: 0 2024-03-10T00:47:47.105 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:47:46 smithi106 ceph-mon[33460]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-03-10T00:47:47.105 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:47:46 smithi106 ceph-mon[33460]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-03-10T00:47:47.105 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:47:46 smithi106 ceph-mon[33460]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-03-10T00:47:47.105 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:47:46 smithi106 ceph-mon[33460]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-03-10T00:47:47.105 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:47:46 smithi106 ceph-mon[33460]: rocksdb: Options.compression_opts.enabled: false 2024-03-10T00:47:47.105 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:47:46 smithi106 ceph-mon[33460]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-03-10T00:47:47.106 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:47:46 smithi106 ceph-mon[33460]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-03-10T00:47:47.106 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:47:46 smithi106 ceph-mon[33460]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-03-10T00:47:47.106 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:47:46 smithi106 ceph-mon[33460]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-03-10T00:47:47.106 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:47:46 smithi106 ceph-mon[33460]: rocksdb: Options.target_file_size_base: 67108864 2024-03-10T00:47:47.106 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:47:46 smithi106 ceph-mon[33460]: rocksdb: Options.target_file_size_multiplier: 1 2024-03-10T00:47:47.106 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:47:46 smithi106 ceph-mon[33460]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-03-10T00:47:47.106 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:47:46 smithi106 ceph-mon[33460]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-03-10T00:47:47.106 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:47:46 smithi106 ceph-mon[33460]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-03-10T00:47:47.106 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:47:46 smithi106 ceph-mon[33460]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-03-10T00:47:47.106 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:47:46 smithi106 ceph-mon[33460]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-03-10T00:47:47.106 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:47:46 smithi106 ceph-mon[33460]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-03-10T00:47:47.106 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:47:46 smithi106 ceph-mon[33460]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-03-10T00:47:47.106 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:47:46 smithi106 ceph-mon[33460]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-03-10T00:47:47.106 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:47:46 smithi106 ceph-mon[33460]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-03-10T00:47:47.107 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:47:46 smithi106 ceph-mon[33460]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-03-10T00:47:47.107 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:47:46 smithi106 ceph-mon[33460]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-03-10T00:47:47.107 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:47:46 smithi106 ceph-mon[33460]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-03-10T00:47:47.107 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:47:46 smithi106 ceph-mon[33460]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-03-10T00:47:47.107 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:47:46 smithi106 ceph-mon[33460]: rocksdb: Options.arena_block_size: 1048576 2024-03-10T00:47:47.107 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:47:46 smithi106 ceph-mon[33460]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-03-10T00:47:47.107 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:47:46 smithi106 ceph-mon[33460]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-03-10T00:47:47.107 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:47:46 smithi106 ceph-mon[33460]: rocksdb: Options.disable_auto_compactions: 0 2024-03-10T00:47:47.107 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:47:46 smithi106 ceph-mon[33460]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-03-10T00:47:47.107 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:47:46 smithi106 ceph-mon[33460]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-03-10T00:47:47.107 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:47:46 smithi106 ceph-mon[33460]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-03-10T00:47:47.107 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:47:46 smithi106 ceph-mon[33460]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-03-10T00:47:47.107 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:47:46 smithi106 ceph-mon[33460]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-03-10T00:47:47.107 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:47:46 smithi106 ceph-mon[33460]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-03-10T00:47:47.108 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:47:46 smithi106 ceph-mon[33460]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-03-10T00:47:47.108 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:47:46 smithi106 ceph-mon[33460]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-03-10T00:47:47.108 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:47:46 smithi106 ceph-mon[33460]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-03-10T00:47:47.108 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:47:46 smithi106 ceph-mon[33460]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-03-10T00:47:47.108 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:47:46 smithi106 ceph-mon[33460]: rocksdb: Options.table_properties_collectors: 2024-03-10T00:47:47.108 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:47:46 smithi106 ceph-mon[33460]: rocksdb: Options.inplace_update_support: 0 2024-03-10T00:47:47.108 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:47:46 smithi106 ceph-mon[33460]: rocksdb: Options.inplace_update_num_locks: 10000 2024-03-10T00:47:47.108 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:47:46 smithi106 ceph-mon[33460]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-03-10T00:47:47.108 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:47:46 smithi106 ceph-mon[33460]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-03-10T00:47:47.108 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:47:46 smithi106 ceph-mon[33460]: rocksdb: Options.memtable_huge_page_size: 0 2024-03-10T00:47:47.108 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:47:46 smithi106 ceph-mon[33460]: rocksdb: Options.bloom_locality: 0 2024-03-10T00:47:47.108 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:47:46 smithi106 ceph-mon[33460]: rocksdb: Options.max_successive_merges: 0 2024-03-10T00:47:47.108 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:47:46 smithi106 ceph-mon[33460]: rocksdb: Options.optimize_filters_for_hits: 0 2024-03-10T00:47:47.109 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:47:46 smithi106 ceph-mon[33460]: rocksdb: Options.paranoid_file_checks: 0 2024-03-10T00:47:47.109 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:47:46 smithi106 ceph-mon[33460]: rocksdb: Options.force_consistency_checks: 1 2024-03-10T00:47:47.109 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:47:46 smithi106 ceph-mon[33460]: rocksdb: Options.report_bg_io_stats: 0 2024-03-10T00:47:47.109 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:47:46 smithi106 ceph-mon[33460]: rocksdb: Options.ttl: 2592000 2024-03-10T00:47:47.109 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:47:46 smithi106 ceph-mon[33460]: rocksdb: Options.periodic_compaction_seconds: 0 2024-03-10T00:47:47.109 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:47:46 smithi106 ceph-mon[33460]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-03-10T00:47:47.109 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:47:46 smithi106 ceph-mon[33460]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-03-10T00:47:47.109 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:47:46 smithi106 ceph-mon[33460]: rocksdb: Options.enable_blob_files: false 2024-03-10T00:47:47.109 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:47:46 smithi106 ceph-mon[33460]: rocksdb: Options.min_blob_size: 0 2024-03-10T00:47:47.109 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:47:46 smithi106 ceph-mon[33460]: rocksdb: Options.blob_file_size: 268435456 2024-03-10T00:47:47.109 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:47:46 smithi106 ceph-mon[33460]: rocksdb: Options.blob_compression_type: NoCompression 2024-03-10T00:47:47.109 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:47:46 smithi106 ceph-mon[33460]: rocksdb: Options.enable_blob_garbage_collection: false 2024-03-10T00:47:47.109 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:47:46 smithi106 ceph-mon[33460]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-03-10T00:47:47.109 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:47:46 smithi106 ceph-mon[33460]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-03-10T00:47:47.110 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:47:46 smithi106 ceph-mon[33460]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-03-10T00:47:47.110 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:47:46 smithi106 ceph-mon[33460]: rocksdb: Options.blob_file_starting_level: 0 2024-03-10T00:47:47.110 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:47:46 smithi106 ceph-mon[33460]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-03-10T00:47:47.110 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:47:46 smithi106 ceph-mon[33460]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-c/store.db/MANIFEST-000005 succeeded,manifest_file_number is 5, next_file_number is 7, last_sequence is 0, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-03-10T00:47:47.110 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:47:46 smithi106 ceph-mon[33460]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 0 2024-03-10T00:47:47.110 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:47:46 smithi106 ceph-mon[33460]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: d3281265-675d-4372-99c2-e70c349b9144 2024-03-10T00:47:47.110 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:47:46 smithi106 ceph-mon[33460]: rocksdb: EVENT_LOG_v1 {"time_micros": 1710031666704955, "job": 1, "event": "recovery_started", "wal_files": [4]} 2024-03-10T00:47:47.110 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:47:46 smithi106 ceph-mon[33460]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #4 mode 2 2024-03-10T00:47:47.110 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:47:46 smithi106 ceph-mon[33460]: rocksdb: EVENT_LOG_v1 {"time_micros": 1710031666705353, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 8, "file_size": 1747, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 1, "largest_seqno": 5, "table_properties": {"data_size": 648, "index_size": 31, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 69, "raw_key_size": 115, "raw_average_key_size": 23, "raw_value_size": 526, "raw_average_value_size": 105, "num_data_blocks": 1, "num_entries": 5, "num_filter_entries": 5, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1710031666, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d3281265-675d-4372-99c2-e70c349b9144", "db_session_id": "V6LMNOAYDPNP337Q668A", "orig_file_number": 8, "seqno_to_time_mapping": "N/A"}} 2024-03-10T00:47:47.110 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:47:46 smithi106 ceph-mon[33460]: rocksdb: EVENT_LOG_v1 {"time_micros": 1710031666705434, "job": 1, "event": "recovery_finished"} 2024-03-10T00:47:47.110 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:47:46 smithi106 ceph-mon[33460]: rocksdb: [db/version_set.cc:5047] Creating manifest 10 2024-03-10T00:47:47.110 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:47:46 smithi106 ceph-mon[33460]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-c/store.db/000004.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-03-10T00:47:47.111 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:47:46 smithi106 ceph-mon[33460]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x55e3e0b86000 2024-03-10T00:47:47.111 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:47:46 smithi106 ceph-mon[33460]: rocksdb: DB pointer 0x55e3e0b70000 2024-03-10T00:47:47.111 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:47:46 smithi106 ceph-mon[33460]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-03-10T00:47:47.111 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:47:46 smithi106 ceph-mon[33460]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-03-10T00:47:47.111 INFO:journalctl@ceph.mon.c.smithi106.stdout: ** DB Stats ** 2024-03-10T00:47:47.111 INFO:journalctl@ceph.mon.c.smithi106.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-03-10T00:47:47.111 INFO:journalctl@ceph.mon.c.smithi106.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-03-10T00:47:47.111 INFO:journalctl@ceph.mon.c.smithi106.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-03-10T00:47:47.111 INFO:journalctl@ceph.mon.c.smithi106.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-03-10T00:47:47.111 INFO:journalctl@ceph.mon.c.smithi106.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-03-10T00:47:47.111 INFO:journalctl@ceph.mon.c.smithi106.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-03-10T00:47:47.111 INFO:journalctl@ceph.mon.c.smithi106.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-03-10T00:47:47.111 INFO:journalctl@ceph.mon.c.smithi106.stdout: 2024-03-10T00:47:47.111 INFO:journalctl@ceph.mon.c.smithi106.stdout: ** Compaction Stats [default] ** 2024-03-10T00:47:47.111 INFO:journalctl@ceph.mon.c.smithi106.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-03-10T00:47:47.112 INFO:journalctl@ceph.mon.c.smithi106.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-03-10T00:47:47.112 INFO:journalctl@ceph.mon.c.smithi106.stdout: L0 1/0 1.71 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 4.4 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-03-10T00:47:47.112 INFO:journalctl@ceph.mon.c.smithi106.stdout: Sum 1/0 1.71 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 4.4 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-03-10T00:47:47.112 INFO:journalctl@ceph.mon.c.smithi106.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 4.4 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-03-10T00:47:47.112 INFO:journalctl@ceph.mon.c.smithi106.stdout: 2024-03-10T00:47:47.112 INFO:journalctl@ceph.mon.c.smithi106.stdout: ** Compaction Stats [default] ** 2024-03-10T00:47:47.112 INFO:journalctl@ceph.mon.c.smithi106.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-03-10T00:47:47.112 INFO:journalctl@ceph.mon.c.smithi106.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-03-10T00:47:47.112 INFO:journalctl@ceph.mon.c.smithi106.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 4.4 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-03-10T00:47:47.112 INFO:journalctl@ceph.mon.c.smithi106.stdout: 2024-03-10T00:47:47.112 INFO:journalctl@ceph.mon.c.smithi106.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-03-10T00:47:47.112 INFO:journalctl@ceph.mon.c.smithi106.stdout: 2024-03-10T00:47:47.112 INFO:journalctl@ceph.mon.c.smithi106.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-03-10T00:47:47.113 INFO:journalctl@ceph.mon.c.smithi106.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-03-10T00:47:47.113 INFO:journalctl@ceph.mon.c.smithi106.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-03-10T00:47:47.113 INFO:journalctl@ceph.mon.c.smithi106.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-03-10T00:47:47.113 INFO:journalctl@ceph.mon.c.smithi106.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-03-10T00:47:47.113 INFO:journalctl@ceph.mon.c.smithi106.stdout: AddFile(Keys): cumulative 0, interval 0 2024-03-10T00:47:47.113 INFO:journalctl@ceph.mon.c.smithi106.stdout: Cumulative compaction: 0.00 GB write, 0.70 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-10T00:47:47.113 INFO:journalctl@ceph.mon.c.smithi106.stdout: Interval compaction: 0.00 GB write, 0.70 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-10T00:47:47.113 INFO:journalctl@ceph.mon.c.smithi106.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-03-10T00:47:47.113 INFO:journalctl@ceph.mon.c.smithi106.stdout: Block cache BinnedLRUCache@0x55e3e0a93090#7 capacity: 512.00 MB usage: 0.22 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 1.2e-05 secs_since: 0 2024-03-10T00:47:47.113 INFO:journalctl@ceph.mon.c.smithi106.stdout: Block cache entry stats(count,size,portion): FilterBlock(1,0.11 KB,2.08616e-05%) IndexBlock(1,0.11 KB,2.08616e-05%) Misc(1,0.00 KB,0%) 2024-03-10T00:47:47.113 INFO:journalctl@ceph.mon.c.smithi106.stdout: 2024-03-10T00:47:47.113 INFO:journalctl@ceph.mon.c.smithi106.stdout: ** File Read Latency Histogram By Level [default] ** 2024-03-10T00:47:47.113 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:47:46 smithi106 ceph-mon[33460]: mon.c does not exist in monmap, will attempt to join an existing cluster 2024-03-10T00:47:47.113 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:47:46 smithi106 ceph-mon[33460]: using public_addrv [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] 2024-03-10T00:47:47.113 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:47:46 smithi106 ceph-mon[33460]: starting mon.c rank -1 at public addrs [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] at bind addrs [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] mon_data /var/lib/ceph/mon/ceph-c fsid 75c37498-de77-11ee-95c7-87774f69a715 2024-03-10T00:47:47.114 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:47:46 smithi106 ceph-mon[33460]: mon.c@-1(???) e0 preinit fsid 75c37498-de77-11ee-95c7-87774f69a715 2024-03-10T00:47:47.114 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:47:46 smithi106 ceph-mon[33460]: mon.c@-1(synchronizing).mds e1 new map 2024-03-10T00:47:47.114 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:47:46 smithi106 ceph-mon[33460]: mon.c@-1(synchronizing).mds e1 print_map 2024-03-10T00:47:47.114 INFO:journalctl@ceph.mon.c.smithi106.stdout: e1 2024-03-10T00:47:47.114 INFO:journalctl@ceph.mon.c.smithi106.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-03-10T00:47:47.114 INFO:journalctl@ceph.mon.c.smithi106.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments} 2024-03-10T00:47:47.114 INFO:journalctl@ceph.mon.c.smithi106.stdout: legacy client fscid: -1 2024-03-10T00:47:47.114 INFO:journalctl@ceph.mon.c.smithi106.stdout: 2024-03-10T00:47:47.114 INFO:journalctl@ceph.mon.c.smithi106.stdout: No filesystems configured 2024-03-10T00:47:47.114 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:47:46 smithi106 ceph-mon[33460]: mon.c@-1(synchronizing).osd e0 _set_cache_ratios kv ratio 0.25 inc ratio 0.375 full ratio 0.375 2024-03-10T00:47:47.114 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:47:46 smithi106 ceph-mon[33460]: mon.c@-1(synchronizing).osd e0 register_cache_with_pcm pcm target: 2147483648 pcm max: 1020054732 pcm min: 134217728 inc_osd_cache size: 1 2024-03-10T00:47:47.114 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:47:46 smithi106 ceph-mon[33460]: mon.c@-1(synchronizing).osd e1 e1: 0 total, 0 up, 0 in 2024-03-10T00:47:47.114 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:47:46 smithi106 ceph-mon[33460]: mon.c@-1(synchronizing).osd e2 e2: 0 total, 0 up, 0 in 2024-03-10T00:47:47.115 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:47:46 smithi106 ceph-mon[33460]: mon.c@-1(synchronizing).osd e3 e3: 0 total, 0 up, 0 in 2024-03-10T00:47:47.115 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:47:46 smithi106 ceph-mon[33460]: mon.c@-1(synchronizing).osd e4 e4: 0 total, 0 up, 0 in 2024-03-10T00:47:47.115 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:47:46 smithi106 ceph-mon[33460]: mon.c@-1(synchronizing).osd e4 crush map has features 3314932999778484224, adjusting msgr requires 2024-03-10T00:47:47.115 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:47:46 smithi106 ceph-mon[33460]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-03-10T00:47:47.115 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:47:46 smithi106 ceph-mon[33460]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-03-10T00:47:47.115 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:47:46 smithi106 ceph-mon[33460]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-03-10T00:47:47.115 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:47:46 smithi106 ceph-mon[33460]: from='client.14178 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi170", "target": ["mon-mgr", ""]}]: dispatch 2024-03-10T00:47:47.115 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:47:46 smithi106 ceph-mon[33460]: Deploying cephadm binary to smithi170 2024-03-10T00:47:47.115 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:47:46 smithi106 ceph-mon[33460]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-10T00:47:47.115 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:47:46 smithi106 ceph-mon[33460]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-10T00:47:47.115 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:47:46 smithi106 ceph-mon[33460]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:47:47.115 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:47:46 smithi106 ceph-mon[33460]: Added host smithi170 2024-03-10T00:47:47.115 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:47:46 smithi106 ceph-mon[33460]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-10T00:47:47.116 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:47:46 smithi106 ceph-mon[33460]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:47:47.116 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:47:46 smithi106 ceph-mon[33460]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-10T00:47:47.116 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:47:46 smithi106 ceph-mon[33460]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:47:47.116 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:47:46 smithi106 ceph-mon[33460]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:47:47.116 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:47:46 smithi106 ceph-mon[33460]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi170", "name": "osd_memory_target"}]: dispatch 2024-03-10T00:47:47.116 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:47:46 smithi106 ceph-mon[33460]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-10T00:47:47.116 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:47:46 smithi106 ceph-mon[33460]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-10T00:47:47.116 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:47:46 smithi106 ceph-mon[33460]: Updating smithi170:/etc/ceph/ceph.conf 2024-03-10T00:47:47.116 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:47:46 smithi106 ceph-mon[33460]: Updating smithi170:/var/lib/ceph/75c37498-de77-11ee-95c7-87774f69a715/config/ceph.conf 2024-03-10T00:47:47.116 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:47:46 smithi106 ceph-mon[33460]: Updating smithi170:/etc/ceph/ceph.client.admin.keyring 2024-03-10T00:47:47.116 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:47:46 smithi106 ceph-mon[33460]: Updating smithi170:/var/lib/ceph/75c37498-de77-11ee-95c7-87774f69a715/config/ceph.client.admin.keyring 2024-03-10T00:47:47.116 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:47:46 smithi106 ceph-mon[33460]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:47:47.116 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:47:46 smithi106 ceph-mon[33460]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:47:47.117 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:47:46 smithi106 ceph-mon[33460]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.agent.smithi170", "caps": []}]: dispatch 2024-03-10T00:47:47.117 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:47:46 smithi106 ceph-mon[33460]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.agent.smithi170", "caps": []}]': finished 2024-03-10T00:47:47.117 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:47:46 smithi106 ceph-mon[33460]: Deploying daemon agent.smithi170 on smithi170 2024-03-10T00:47:47.117 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:47:46 smithi106 ceph-mon[33460]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-10T00:47:47.117 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:47:46 smithi106 ceph-mon[33460]: from='client.14180 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-10T00:47:47.117 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:47:46 smithi106 ceph-mon[33460]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:47:47.117 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:47:46 smithi106 ceph-mon[33460]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:47:47.117 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:47:46 smithi106 ceph-mon[33460]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:47:47.117 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:47:46 smithi106 ceph-mon[33460]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-10T00:47:47.117 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:47:46 smithi106 ceph-mon[33460]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-10T00:47:47.117 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:47:46 smithi106 ceph-mon[33460]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-10T00:47:47.117 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:47:46 smithi106 ceph-mon[33460]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:47:47.117 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:47:46 smithi106 ceph-mon[33460]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-10T00:47:47.117 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:47:46 smithi106 ceph-mon[33460]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-10T00:47:47.117 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:47:46 smithi106 ceph-mon[33460]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-10T00:47:47.118 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:47:46 smithi106 ceph-mon[33460]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:47:47.118 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:47:46 smithi106 ceph-mon[33460]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:47:47.118 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:47:46 smithi106 ceph-mon[33460]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:47:47.118 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:47:46 smithi106 ceph-mon[33460]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:47:47.118 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:47:46 smithi106 ceph-mon[33460]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-10T00:47:47.118 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:47:46 smithi106 ceph-mon[33460]: from='client.? 172.21.15.106:0/3394949640' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-03-10T00:47:47.118 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:47:46 smithi106 ceph-mon[33460]: from='client.? 172.21.15.106:0/3394949640' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-03-10T00:47:47.118 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:47:46 smithi106 ceph-mon[33460]: osdmap e4: 0 total, 0 up, 0 in 2024-03-10T00:47:47.118 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:47:46 smithi106 ceph-mon[33460]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:47:47.118 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:47:46 smithi106 ceph-mon[33460]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:47:47.118 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:47:46 smithi106 ceph-mon[33460]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-10T00:47:47.118 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:47:46 smithi106 ceph-mon[33460]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:47:47.118 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:47:46 smithi106 ceph-mon[33460]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:47:47.118 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:47:46 smithi106 ceph-mon[33460]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:47:47.118 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:47:46 smithi106 ceph-mon[33460]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:47:47.119 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:47:46 smithi106 ceph-mon[33460]: from='client.14184 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi106:172.21.15.106=a;smithi106:[v2:172.21.15.106:3301,v1:172.21.15.106:6790]=c;smithi170:172.21.15.170=b", "target": ["mon-mgr", ""]}]: dispatch 2024-03-10T00:47:47.119 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:47:46 smithi106 ceph-mon[33460]: Saving service mon spec with placement smithi106:172.21.15.106=a;smithi106:[v2:172.21.15.106:3301,v1:172.21.15.106:6790]=c;smithi170:172.21.15.170=b;count:3 2024-03-10T00:47:47.119 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:47:46 smithi106 ceph-mon[33460]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:47:47.119 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:47:46 smithi106 ceph-mon[33460]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-10T00:47:47.119 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:47:46 smithi106 ceph-mon[33460]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-10T00:47:47.119 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:47:46 smithi106 ceph-mon[33460]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-10T00:47:47.119 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:47:46 smithi106 ceph-mon[33460]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:47:47.119 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:47:46 smithi106 ceph-mon[33460]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-10T00:47:47.119 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:47:46 smithi106 ceph-mon[33460]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-10T00:47:47.119 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:47:46 smithi106 ceph-mon[33460]: Deploying daemon mon.b on smithi170 2024-03-10T00:47:47.119 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:47:46 smithi106 ceph-mon[33460]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-10T00:47:47.119 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:47:46 smithi106 ceph-mon[33460]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-10T00:47:47.119 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:47:46 smithi106 ceph-mon[33460]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-10T00:47:47.119 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:47:46 smithi106 ceph-mon[33460]: from='client.? 172.21.15.170:0/1931631842' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-10T00:47:47.120 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:47:46 smithi106 ceph-mon[33460]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-10T00:47:47.120 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:47:46 smithi106 ceph-mon[33460]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-10T00:47:47.120 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:47:46 smithi106 ceph-mon[33460]: mon.a calling monitor election 2024-03-10T00:47:47.120 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:47:46 smithi106 ceph-mon[33460]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-10T00:47:47.120 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:47:46 smithi106 ceph-mon[33460]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-10T00:47:47.120 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:47:46 smithi106 ceph-mon[33460]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-10T00:47:47.120 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:47:46 smithi106 ceph-mon[33460]: mon.b calling monitor election 2024-03-10T00:47:47.120 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:47:46 smithi106 ceph-mon[33460]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-10T00:47:47.120 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:47:46 smithi106 ceph-mon[33460]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-10T00:47:47.120 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:47:46 smithi106 ceph-mon[33460]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-10T00:47:47.120 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:47:46 smithi106 ceph-mon[33460]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-10T00:47:47.120 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:47:46 smithi106 ceph-mon[33460]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-10T00:47:47.120 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:47:46 smithi106 ceph-mon[33460]: mon.a is new leader, mons a,b in quorum (ranks 0,1) 2024-03-10T00:47:47.120 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:47:46 smithi106 ceph-mon[33460]: monmap e2: 2 mons at {a=[v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0],b=[v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-10T00:47:47.120 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:47:46 smithi106 ceph-mon[33460]: fsmap 2024-03-10T00:47:47.120 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:47:46 smithi106 ceph-mon[33460]: osdmap e4: 0 total, 0 up, 0 in 2024-03-10T00:47:47.121 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:47:46 smithi106 ceph-mon[33460]: mgrmap e13: y(active, since 42s) 2024-03-10T00:47:47.121 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:47:46 smithi106 ceph-mon[33460]: overall HEALTH_OK 2024-03-10T00:47:47.121 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:47:46 smithi106 ceph-mon[33460]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:47:47.121 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:47:46 smithi106 ceph-mon[33460]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:47:47.121 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:47:46 smithi106 ceph-mon[33460]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:47:47.121 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:47:46 smithi106 ceph-mon[33460]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-10T00:47:47.121 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:47:46 smithi106 ceph-mon[33460]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-10T00:47:47.121 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:47:46 smithi106 ceph-mon[33460]: Deploying daemon mon.c on smithi106 2024-03-10T00:47:47.121 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:47:46 smithi106 ceph-mon[33460]: from='client.? 172.21.15.170:0/919666768' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-10T00:47:47.121 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:47:46 smithi106 ceph-mon[33460]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-10T00:47:47.121 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:47:46 smithi106 ceph-mon[33460]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-10T00:47:47.121 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:47:46 smithi106 ceph-mon[33460]: mon.c@-1(synchronizing).paxosservice(auth 1..5) refresh upgraded, format 0 -> 3 2024-03-10T00:47:47.121 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:47:46 smithi106 bash[33422]: 2411e393082d5ae1f677c7a188bf2ee903ed532e4f6db2cd90cc776ace2c1493 2024-03-10T00:47:47.121 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:47:46 smithi106 systemd[1]: Started Ceph mon.c for 75c37498-de77-11ee-95c7-87774f69a715. 2024-03-10T00:47:51.956 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:47:51 smithi106 ceph-mon[25742]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-10T00:47:51.956 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:47:51 smithi106 ceph-mon[25742]: mon.a calling monitor election 2024-03-10T00:47:51.956 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:47:51 smithi106 ceph-mon[25742]: mon.b calling monitor election 2024-03-10T00:47:51.956 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:47:51 smithi106 ceph-mon[25742]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-10T00:47:51.956 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:47:51 smithi106 ceph-mon[25742]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-10T00:47:51.956 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:47:51 smithi106 ceph-mon[25742]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-10T00:47:51.957 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:47:51 smithi106 ceph-mon[25742]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-10T00:47:51.957 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:47:51 smithi106 ceph-mon[25742]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-10T00:47:51.957 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:47:51 smithi106 ceph-mon[25742]: mon.c calling monitor election 2024-03-10T00:47:51.957 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:47:51 smithi106 ceph-mon[25742]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-10T00:47:51.957 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:47:51 smithi106 ceph-mon[25742]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-10T00:47:51.957 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:47:51 smithi106 ceph-mon[25742]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-10T00:47:51.957 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:47:51 smithi106 ceph-mon[25742]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-10T00:47:51.957 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:47:51 smithi106 ceph-mon[25742]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-10T00:47:51.957 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:47:51 smithi106 ceph-mon[25742]: mon.a is new leader, mons a,b,c in quorum (ranks 0,1,2) 2024-03-10T00:47:51.957 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:47:51 smithi106 ceph-mon[25742]: monmap e3: 3 mons at {a=[v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0],b=[v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0],c=[v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-10T00:47:51.957 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:47:51 smithi106 ceph-mon[25742]: fsmap 2024-03-10T00:47:51.957 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:47:51 smithi106 ceph-mon[25742]: osdmap e4: 0 total, 0 up, 0 in 2024-03-10T00:47:51.958 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:47:51 smithi106 ceph-mon[25742]: mgrmap e13: y(active, since 51s) 2024-03-10T00:47:51.958 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:47:51 smithi106 ceph-mon[25742]: overall HEALTH_OK 2024-03-10T00:47:51.958 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:47:51 smithi106 ceph-mon[25742]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:47:51.958 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:47:51 smithi106 ceph-mon[25742]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:47:51.958 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:47:51 smithi106 ceph-mon[25742]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:47:51.958 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:47:51 smithi106 ceph-mon[25742]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:47:51.958 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:47:51 smithi106 ceph-mon[25742]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:47:51.958 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:47:51 smithi106 ceph-mon[33460]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-10T00:47:51.958 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:47:51 smithi106 ceph-mon[33460]: mon.a calling monitor election 2024-03-10T00:47:51.959 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:47:51 smithi106 ceph-mon[33460]: mon.b calling monitor election 2024-03-10T00:47:51.959 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:47:51 smithi106 ceph-mon[33460]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-10T00:47:51.959 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:47:51 smithi106 ceph-mon[33460]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-10T00:47:51.959 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:47:51 smithi106 ceph-mon[33460]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-10T00:47:51.959 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:47:51 smithi106 ceph-mon[33460]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-10T00:47:51.959 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:47:51 smithi106 ceph-mon[33460]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-10T00:47:51.959 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:47:51 smithi106 ceph-mon[33460]: mon.c calling monitor election 2024-03-10T00:47:51.959 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:47:51 smithi106 ceph-mon[33460]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-10T00:47:51.959 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:47:51 smithi106 ceph-mon[33460]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-10T00:47:51.959 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:47:51 smithi106 ceph-mon[33460]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-10T00:47:51.959 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:47:51 smithi106 ceph-mon[33460]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-10T00:47:51.959 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:47:51 smithi106 ceph-mon[33460]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-10T00:47:51.959 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:47:51 smithi106 ceph-mon[33460]: mon.a is new leader, mons a,b,c in quorum (ranks 0,1,2) 2024-03-10T00:47:51.959 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:47:51 smithi106 ceph-mon[33460]: monmap e3: 3 mons at {a=[v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0],b=[v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0],c=[v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-10T00:47:51.959 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:47:51 smithi106 ceph-mon[33460]: fsmap 2024-03-10T00:47:51.960 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:47:51 smithi106 ceph-mon[33460]: osdmap e4: 0 total, 0 up, 0 in 2024-03-10T00:47:51.960 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:47:51 smithi106 ceph-mon[33460]: mgrmap e13: y(active, since 51s) 2024-03-10T00:47:51.960 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:47:51 smithi106 ceph-mon[33460]: overall HEALTH_OK 2024-03-10T00:47:51.960 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:47:51 smithi106 ceph-mon[33460]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:47:51.960 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:47:51 smithi106 ceph-mon[33460]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:47:51.960 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:47:51 smithi106 ceph-mon[33460]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:47:51.960 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:47:51 smithi106 ceph-mon[33460]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:47:51.960 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:47:51 smithi106 ceph-mon[33460]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:47:52.275 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:47:51 smithi170 ceph-mon[27620]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-10T00:47:52.275 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:47:51 smithi170 ceph-mon[27620]: mon.a calling monitor election 2024-03-10T00:47:52.275 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:47:51 smithi170 ceph-mon[27620]: mon.b calling monitor election 2024-03-10T00:47:52.275 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:47:51 smithi170 ceph-mon[27620]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-10T00:47:52.275 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:47:51 smithi170 ceph-mon[27620]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-10T00:47:52.275 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:47:51 smithi170 ceph-mon[27620]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-10T00:47:52.275 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:47:51 smithi170 ceph-mon[27620]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-10T00:47:52.275 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:47:51 smithi170 ceph-mon[27620]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-10T00:47:52.275 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:47:51 smithi170 ceph-mon[27620]: mon.c calling monitor election 2024-03-10T00:47:52.276 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:47:51 smithi170 ceph-mon[27620]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-10T00:47:52.276 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:47:51 smithi170 ceph-mon[27620]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-10T00:47:52.276 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:47:51 smithi170 ceph-mon[27620]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-10T00:47:52.276 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:47:51 smithi170 ceph-mon[27620]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-10T00:47:52.276 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:47:51 smithi170 ceph-mon[27620]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-10T00:47:52.276 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:47:51 smithi170 ceph-mon[27620]: mon.a is new leader, mons a,b,c in quorum (ranks 0,1,2) 2024-03-10T00:47:52.276 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:47:51 smithi170 ceph-mon[27620]: monmap e3: 3 mons at {a=[v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0],b=[v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0],c=[v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-10T00:47:52.276 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:47:51 smithi170 ceph-mon[27620]: fsmap 2024-03-10T00:47:52.276 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:47:51 smithi170 ceph-mon[27620]: osdmap e4: 0 total, 0 up, 0 in 2024-03-10T00:47:52.276 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:47:51 smithi170 ceph-mon[27620]: mgrmap e13: y(active, since 51s) 2024-03-10T00:47:52.276 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:47:51 smithi170 ceph-mon[27620]: overall HEALTH_OK 2024-03-10T00:47:52.276 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:47:51 smithi170 ceph-mon[27620]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:47:52.276 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:47:51 smithi170 ceph-mon[27620]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:47:52.276 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:47:51 smithi170 ceph-mon[27620]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:47:52.277 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:47:51 smithi170 ceph-mon[27620]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:47:52.277 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:47:51 smithi170 ceph-mon[27620]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:47:53.089 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:47:52 smithi106 ceph-mon[25742]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:47:53.089 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:47:52 smithi106 ceph-mon[25742]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:47:53.089 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:47:52 smithi106 ceph-mon[25742]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-10T00:47:53.089 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:47:52 smithi106 ceph-mon[25742]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-10T00:47:53.089 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:47:52 smithi106 ceph-mon[25742]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-10T00:47:53.089 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:47:52 smithi106 ceph-mon[25742]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:47:53.090 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:47:52 smithi106 ceph-mon[25742]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:47:53.090 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:47:52 smithi106 ceph-mon[25742]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:47:53.090 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:47:52 smithi106 ceph-mon[25742]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:47:53.090 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:47:52 smithi106 ceph-mon[25742]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:47:53.090 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:47:52 smithi106 ceph-mon[25742]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:47:53.090 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:47:52 smithi106 ceph-mon[25742]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:47:53.090 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:47:52 smithi106 ceph-mon[25742]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:47:53.090 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:47:52 smithi106 ceph-mon[25742]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-10T00:47:53.090 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:47:52 smithi106 ceph-mon[25742]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-10T00:47:53.090 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:47:52 smithi106 ceph-mon[25742]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-10T00:47:53.090 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:47:52 smithi106 ceph-mon[25742]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-10T00:47:53.091 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:47:52 smithi106 ceph-mon[33460]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:47:53.091 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:47:52 smithi106 ceph-mon[33460]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:47:53.091 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:47:52 smithi106 ceph-mon[33460]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-10T00:47:53.091 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:47:52 smithi106 ceph-mon[33460]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-10T00:47:53.091 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:47:52 smithi106 ceph-mon[33460]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-10T00:47:53.091 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:47:52 smithi106 ceph-mon[33460]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:47:53.091 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:47:52 smithi106 ceph-mon[33460]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:47:53.091 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:47:52 smithi106 ceph-mon[33460]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:47:53.091 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:47:52 smithi106 ceph-mon[33460]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:47:53.091 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:47:52 smithi106 ceph-mon[33460]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:47:53.091 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:47:52 smithi106 ceph-mon[33460]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:47:53.091 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:47:52 smithi106 ceph-mon[33460]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:47:53.091 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:47:52 smithi106 ceph-mon[33460]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:47:53.092 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:47:52 smithi106 ceph-mon[33460]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-10T00:47:53.092 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:47:52 smithi106 ceph-mon[33460]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-10T00:47:53.092 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:47:52 smithi106 ceph-mon[33460]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-10T00:47:53.092 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:47:52 smithi106 ceph-mon[33460]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-10T00:47:53.096 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:47:52 smithi170 ceph-mon[27620]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:47:53.096 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:47:52 smithi170 ceph-mon[27620]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:47:53.096 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:47:52 smithi170 ceph-mon[27620]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-10T00:47:53.096 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:47:52 smithi170 ceph-mon[27620]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-10T00:47:53.096 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:47:52 smithi170 ceph-mon[27620]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-10T00:47:53.097 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:47:52 smithi170 ceph-mon[27620]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:47:53.097 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:47:52 smithi170 ceph-mon[27620]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:47:53.097 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:47:52 smithi170 ceph-mon[27620]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:47:53.097 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:47:52 smithi170 ceph-mon[27620]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:47:53.097 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:47:52 smithi170 ceph-mon[27620]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:47:53.097 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:47:52 smithi170 ceph-mon[27620]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:47:53.097 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:47:52 smithi170 ceph-mon[27620]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:47:53.097 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:47:52 smithi170 ceph-mon[27620]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:47:53.097 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:47:52 smithi170 ceph-mon[27620]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-10T00:47:53.097 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:47:52 smithi170 ceph-mon[27620]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-10T00:47:53.097 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:47:52 smithi170 ceph-mon[27620]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-10T00:47:53.097 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:47:52 smithi170 ceph-mon[27620]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-10T00:47:53.208 INFO:teuthology.orchestra.run.smithi170.stdout: 2024-03-10T00:47:53.208 INFO:teuthology.orchestra.run.smithi170.stdout:{"epoch":3,"fsid":"75c37498-de77-11ee-95c7-87774f69a715","modified":"2024-03-10T00:47:46.726311Z","created":"2024-03-10T00:45:55.067658Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:3300","nonce":0},{"type":"v1","addr":"172.21.15.106:6789","nonce":0}]},"addr":"172.21.15.106:6789/0","public_addr":"172.21.15.106:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:3300","nonce":0},{"type":"v1","addr":"172.21.15.170:6789","nonce":0}]},"addr":"172.21.15.170:6789/0","public_addr":"172.21.15.170:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:3301","nonce":0},{"type":"v1","addr":"172.21.15.106:6790","nonce":0}]},"addr":"172.21.15.106:6790/0","public_addr":"172.21.15.106:6790/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1,2]} 2024-03-10T00:47:53.209 INFO:teuthology.orchestra.run.smithi170.stderr:dumped monmap epoch 3 2024-03-10T00:47:54.093 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:47:53 smithi170 ceph-mon[27620]: Updating smithi106:/etc/ceph/ceph.conf 2024-03-10T00:47:54.093 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:47:53 smithi170 ceph-mon[27620]: Updating smithi170:/etc/ceph/ceph.conf 2024-03-10T00:47:54.093 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:47:53 smithi170 ceph-mon[27620]: Updating smithi106:/var/lib/ceph/75c37498-de77-11ee-95c7-87774f69a715/config/ceph.conf 2024-03-10T00:47:54.094 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:47:53 smithi170 ceph-mon[27620]: Updating smithi170:/var/lib/ceph/75c37498-de77-11ee-95c7-87774f69a715/config/ceph.conf 2024-03-10T00:47:54.094 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:47:53 smithi170 ceph-mon[27620]: Metadata not up to date on all hosts. Skipping non agent specs 2024-03-10T00:47:54.094 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:47:53 smithi170 ceph-mon[27620]: Reconfiguring mon.a (unknown last config time)... 2024-03-10T00:47:54.094 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:47:53 smithi170 ceph-mon[27620]: Reconfiguring daemon mon.a on smithi106 2024-03-10T00:47:54.094 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:47:53 smithi170 ceph-mon[27620]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-10T00:47:54.094 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:47:53 smithi170 ceph-mon[27620]: from='client.? 172.21.15.170:0/3493771275' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-10T00:47:54.094 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:47:53 smithi170 ceph-mon[27620]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:47:54.094 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:47:53 smithi170 ceph-mon[27620]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:47:54.094 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:47:53 smithi170 ceph-mon[27620]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:47:54.179 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:47:53 smithi106 ceph-mon[25742]: Updating smithi106:/etc/ceph/ceph.conf 2024-03-10T00:47:54.179 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:47:53 smithi106 ceph-mon[25742]: Updating smithi170:/etc/ceph/ceph.conf 2024-03-10T00:47:54.179 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:47:53 smithi106 ceph-mon[25742]: Updating smithi106:/var/lib/ceph/75c37498-de77-11ee-95c7-87774f69a715/config/ceph.conf 2024-03-10T00:47:54.179 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:47:53 smithi106 ceph-mon[25742]: Updating smithi170:/var/lib/ceph/75c37498-de77-11ee-95c7-87774f69a715/config/ceph.conf 2024-03-10T00:47:54.179 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:47:53 smithi106 ceph-mon[25742]: Metadata not up to date on all hosts. Skipping non agent specs 2024-03-10T00:47:54.180 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:47:53 smithi106 ceph-mon[25742]: Reconfiguring mon.a (unknown last config time)... 2024-03-10T00:47:54.180 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:47:53 smithi106 ceph-mon[25742]: Reconfiguring daemon mon.a on smithi106 2024-03-10T00:47:54.180 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:47:53 smithi106 ceph-mon[25742]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-10T00:47:54.180 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:47:53 smithi106 ceph-mon[25742]: from='client.? 172.21.15.170:0/3493771275' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-10T00:47:54.180 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:47:53 smithi106 ceph-mon[25742]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:47:54.180 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:47:53 smithi106 ceph-mon[25742]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:47:54.180 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:47:53 smithi106 ceph-mon[25742]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:47:54.180 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:47:53 smithi106 ceph-mon[33460]: Updating smithi106:/etc/ceph/ceph.conf 2024-03-10T00:47:54.180 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:47:53 smithi106 ceph-mon[33460]: Updating smithi170:/etc/ceph/ceph.conf 2024-03-10T00:47:54.180 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:47:53 smithi106 ceph-mon[33460]: Updating smithi106:/var/lib/ceph/75c37498-de77-11ee-95c7-87774f69a715/config/ceph.conf 2024-03-10T00:47:54.180 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:47:53 smithi106 ceph-mon[33460]: Updating smithi170:/var/lib/ceph/75c37498-de77-11ee-95c7-87774f69a715/config/ceph.conf 2024-03-10T00:47:54.181 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:47:53 smithi106 ceph-mon[33460]: Metadata not up to date on all hosts. Skipping non agent specs 2024-03-10T00:47:54.181 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:47:53 smithi106 ceph-mon[33460]: Reconfiguring mon.a (unknown last config time)... 2024-03-10T00:47:54.181 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:47:53 smithi106 ceph-mon[33460]: Reconfiguring daemon mon.a on smithi106 2024-03-10T00:47:54.181 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:47:53 smithi106 ceph-mon[33460]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-10T00:47:54.181 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:47:53 smithi106 ceph-mon[33460]: from='client.? 172.21.15.170:0/3493771275' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-10T00:47:54.181 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:47:53 smithi106 ceph-mon[33460]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:47:54.181 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:47:53 smithi106 ceph-mon[33460]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:47:54.181 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:47:53 smithi106 ceph-mon[33460]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:47:54.583 INFO:tasks.cephadm:Generating final ceph.conf file... 2024-03-10T00:47:54.583 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:831bc188fa69f143bbba69ac76119e784500e8e3 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 75c37498-de77-11ee-95c7-87774f69a715 -- ceph config generate-minimal-conf 2024-03-10T00:47:56.005 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:47:55 smithi106 ceph-mon[33460]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-10T00:47:56.005 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:47:55 smithi106 ceph-mon[33460]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:47:56.005 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:47:55 smithi106 ceph-mon[33460]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:47:56.005 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:47:55 smithi106 ceph-mon[33460]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:47:56.005 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:47:55 smithi106 ceph-mon[33460]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:47:56.005 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:47:55 smithi106 ceph-mon[33460]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:47:56.005 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:47:55 smithi106 ceph-mon[25742]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-10T00:47:56.005 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:47:55 smithi106 ceph-mon[25742]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:47:56.006 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:47:55 smithi106 ceph-mon[25742]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:47:56.006 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:47:55 smithi106 ceph-mon[25742]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:47:56.006 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:47:55 smithi106 ceph-mon[25742]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:47:56.006 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:47:55 smithi106 ceph-mon[25742]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:47:56.054 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/75c37498-de77-11ee-95c7-87774f69a715/mon.a/config 2024-03-10T00:47:56.274 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:47:55 smithi170 ceph-mon[27620]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-10T00:47:56.275 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:47:55 smithi170 ceph-mon[27620]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:47:56.275 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:47:55 smithi170 ceph-mon[27620]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:47:56.275 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:47:55 smithi170 ceph-mon[27620]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:47:56.275 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:47:55 smithi170 ceph-mon[27620]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:47:56.275 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:47:55 smithi170 ceph-mon[27620]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:47:57.089 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:47:57 smithi106 ceph-mon[33460]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:47:57.089 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:47:57 smithi106 ceph-mon[33460]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:47:57.089 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:47:57 smithi106 ceph-mon[33460]: Reconfiguring mon.b (monmap changed)... 2024-03-10T00:47:57.090 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:47:57 smithi106 ceph-mon[33460]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-10T00:47:57.090 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:47:57 smithi106 ceph-mon[33460]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-10T00:47:57.090 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:47:57 smithi106 ceph-mon[33460]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-10T00:47:57.090 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:47:57 smithi106 ceph-mon[33460]: Reconfiguring daemon mon.b on smithi170 2024-03-10T00:47:57.090 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:47:57 smithi106 ceph-mon[33460]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:47:57.090 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:47:57 smithi106 ceph-mon[33460]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:47:57.090 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:47:57 smithi106 ceph-mon[33460]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:47:57.090 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:47:57 smithi106 ceph-mon[33460]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-10T00:47:57.090 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:47:57 smithi106 ceph-mon[25742]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:47:57.090 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:47:57 smithi106 ceph-mon[25742]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:47:57.090 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:47:57 smithi106 ceph-mon[25742]: Reconfiguring mon.b (monmap changed)... 2024-03-10T00:47:57.091 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:47:57 smithi106 ceph-mon[25742]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-10T00:47:57.091 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:47:57 smithi106 ceph-mon[25742]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-10T00:47:57.091 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:47:57 smithi106 ceph-mon[25742]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-10T00:47:57.091 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:47:57 smithi106 ceph-mon[25742]: Reconfiguring daemon mon.b on smithi170 2024-03-10T00:47:57.091 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:47:57 smithi106 ceph-mon[25742]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:47:57.091 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:47:57 smithi106 ceph-mon[25742]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:47:57.091 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:47:57 smithi106 ceph-mon[25742]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:47:57.091 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:47:57 smithi106 ceph-mon[25742]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-10T00:47:57.275 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:47:57 smithi170 ceph-mon[27620]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:47:57.275 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:47:57 smithi170 ceph-mon[27620]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:47:57.275 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:47:57 smithi170 ceph-mon[27620]: Reconfiguring mon.b (monmap changed)... 2024-03-10T00:47:57.275 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:47:57 smithi170 ceph-mon[27620]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-10T00:47:57.275 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:47:57 smithi170 ceph-mon[27620]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-10T00:47:57.275 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:47:57 smithi170 ceph-mon[27620]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-10T00:47:57.275 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:47:57 smithi170 ceph-mon[27620]: Reconfiguring daemon mon.b on smithi170 2024-03-10T00:47:57.275 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:47:57 smithi170 ceph-mon[27620]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:47:57.275 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:47:57 smithi170 ceph-mon[27620]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:47:57.275 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:47:57 smithi170 ceph-mon[27620]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:47:57.276 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:47:57 smithi170 ceph-mon[27620]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-10T00:47:58.805 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:47:58 smithi106 ceph-mon[25742]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:47:58.805 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:47:58 smithi106 ceph-mon[25742]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:47:58.805 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:47:58 smithi106 ceph-mon[25742]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-10T00:47:58.805 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:47:58 smithi106 ceph-mon[25742]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-10T00:47:58.805 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:47:58 smithi106 ceph-mon[25742]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-10T00:47:58.805 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:47:58 smithi106 ceph-mon[25742]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:47:58.806 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:47:58 smithi106 ceph-mon[25742]: Reconfiguring mon.c (monmap changed)... 2024-03-10T00:47:58.806 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:47:58 smithi106 ceph-mon[25742]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-10T00:47:58.806 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:47:58 smithi106 ceph-mon[25742]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-10T00:47:58.806 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:47:58 smithi106 ceph-mon[25742]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-10T00:47:58.806 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:47:58 smithi106 ceph-mon[25742]: Reconfiguring daemon mon.c on smithi106 2024-03-10T00:47:58.806 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:47:58 smithi106 ceph-mon[25742]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:47:58.806 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:47:58 smithi106 ceph-mon[25742]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:47:58.806 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:47:58 smithi106 ceph-mon[25742]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:47:58.806 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:47:58 smithi106 ceph-mon[33460]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:47:58.806 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:47:58 smithi106 ceph-mon[33460]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:47:58.806 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:47:58 smithi106 ceph-mon[33460]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-10T00:47:58.806 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:47:58 smithi106 ceph-mon[33460]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-10T00:47:58.807 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:47:58 smithi106 ceph-mon[33460]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-10T00:47:58.807 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:47:58 smithi106 ceph-mon[33460]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:47:58.807 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:47:58 smithi106 ceph-mon[33460]: Reconfiguring mon.c (monmap changed)... 2024-03-10T00:47:58.807 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:47:58 smithi106 ceph-mon[33460]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-10T00:47:58.807 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:47:58 smithi106 ceph-mon[33460]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-10T00:47:58.807 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:47:58 smithi106 ceph-mon[33460]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-10T00:47:58.807 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:47:58 smithi106 ceph-mon[33460]: Reconfiguring daemon mon.c on smithi106 2024-03-10T00:47:58.807 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:47:58 smithi106 ceph-mon[33460]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:47:58.807 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:47:58 smithi106 ceph-mon[33460]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:47:58.807 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:47:58 smithi106 ceph-mon[33460]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:47:58.934 INFO:teuthology.orchestra.run.smithi106.stdout:# minimal ceph.conf for 75c37498-de77-11ee-95c7-87774f69a715 2024-03-10T00:47:58.935 INFO:teuthology.orchestra.run.smithi106.stdout:[global] 2024-03-10T00:47:58.935 INFO:teuthology.orchestra.run.smithi106.stdout: fsid = 75c37498-de77-11ee-95c7-87774f69a715 2024-03-10T00:47:58.935 INFO:teuthology.orchestra.run.smithi106.stdout: mon_host = [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] 2024-03-10T00:47:59.025 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:47:58 smithi170 ceph-mon[27620]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:47:59.025 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:47:58 smithi170 ceph-mon[27620]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:47:59.025 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:47:58 smithi170 ceph-mon[27620]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-10T00:47:59.025 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:47:58 smithi170 ceph-mon[27620]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-10T00:47:59.025 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:47:58 smithi170 ceph-mon[27620]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-10T00:47:59.025 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:47:58 smithi170 ceph-mon[27620]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:47:59.025 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:47:58 smithi170 ceph-mon[27620]: Reconfiguring mon.c (monmap changed)... 2024-03-10T00:47:59.025 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:47:58 smithi170 ceph-mon[27620]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-10T00:47:59.025 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:47:58 smithi170 ceph-mon[27620]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-10T00:47:59.025 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:47:58 smithi170 ceph-mon[27620]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-10T00:47:59.026 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:47:58 smithi170 ceph-mon[27620]: Reconfiguring daemon mon.c on smithi106 2024-03-10T00:47:59.026 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:47:58 smithi170 ceph-mon[27620]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:47:59.026 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:47:58 smithi170 ceph-mon[27620]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:47:59.026 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:47:58 smithi170 ceph-mon[27620]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:47:59.585 INFO:tasks.cephadm:Distributing (final) config and client.admin keyring... 2024-03-10T00:47:59.585 DEBUG:teuthology.orchestra.run.smithi106:> set -ex 2024-03-10T00:47:59.585 DEBUG:teuthology.orchestra.run.smithi106:> sudo dd of=/etc/ceph/ceph.conf 2024-03-10T00:47:59.614 DEBUG:teuthology.orchestra.run.smithi106:> set -ex 2024-03-10T00:47:59.614 DEBUG:teuthology.orchestra.run.smithi106:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-03-10T00:47:59.682 DEBUG:teuthology.orchestra.run.smithi170:> set -ex 2024-03-10T00:47:59.682 DEBUG:teuthology.orchestra.run.smithi170:> sudo dd of=/etc/ceph/ceph.conf 2024-03-10T00:47:59.717 DEBUG:teuthology.orchestra.run.smithi170:> set -ex 2024-03-10T00:47:59.718 DEBUG:teuthology.orchestra.run.smithi170:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-03-10T00:47:59.787 INFO:tasks.cephadm:Adding mgr.y on smithi106 2024-03-10T00:47:59.787 INFO:tasks.cephadm:Adding mgr.x on smithi170 2024-03-10T00:47:59.787 DEBUG:teuthology.orchestra.run.smithi170:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:831bc188fa69f143bbba69ac76119e784500e8e3 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 75c37498-de77-11ee-95c7-87774f69a715 -- ceph orch apply mgr '2;smithi106=y;smithi170=x' 2024-03-10T00:47:59.838 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:47:59 smithi106 ceph-mon[25742]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-10T00:47:59.839 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:47:59 smithi106 ceph-mon[25742]: from='client.? 172.21.15.106:0/2262532405' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-10T00:47:59.839 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:47:59 smithi106 ceph-mon[33460]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-10T00:47:59.839 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:47:59 smithi106 ceph-mon[33460]: from='client.? 172.21.15.106:0/2262532405' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-10T00:48:00.025 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:47:59 smithi170 ceph-mon[27620]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-10T00:48:00.025 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:47:59 smithi170 ceph-mon[27620]: from='client.? 172.21.15.106:0/2262532405' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-10T00:48:00.061 INFO:teuthology.orchestra.run.smithi170.stderr:Inferring config /var/lib/ceph/75c37498-de77-11ee-95c7-87774f69a715/mon.b/config 2024-03-10T00:48:01.524 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:48:01 smithi170 ceph-mon[27620]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:48:01.525 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:48:01 smithi170 ceph-mon[27620]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:48:01.525 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:48:01 smithi170 ceph-mon[27620]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-10T00:48:01.790 INFO:teuthology.orchestra.run.smithi170.stdout:Scheduled mgr update... 2024-03-10T00:48:01.839 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:48:01 smithi106 ceph-mon[25742]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:48:01.839 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:48:01 smithi106 ceph-mon[25742]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:48:01.839 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:48:01 smithi106 ceph-mon[25742]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-10T00:48:01.839 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:48:01 smithi106 ceph-mon[33460]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:48:01.839 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:48:01 smithi106 ceph-mon[33460]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:48:01.839 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:48:01 smithi106 ceph-mon[33460]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-10T00:48:02.321 DEBUG:teuthology.orchestra.run.smithi170:mgr.x> sudo journalctl -f -n 0 -u ceph-75c37498-de77-11ee-95c7-87774f69a715@mgr.x.service 2024-03-10T00:48:02.324 INFO:tasks.cephadm:Deploying OSDs... 2024-03-10T00:48:02.324 DEBUG:teuthology.orchestra.run.smithi106:> set -ex 2024-03-10T00:48:02.324 DEBUG:teuthology.orchestra.run.smithi106:> dd if=/scratch_devs of=/dev/stdout 2024-03-10T00:48:02.343 DEBUG:teuthology.misc:devs=['/dev/nvme1n1', '/dev/nvme2n1', '/dev/nvme3n1', '/dev/nvme4n1'] 2024-03-10T00:48:02.344 DEBUG:teuthology.orchestra.run.smithi106:> stat /dev/nvme1n1 2024-03-10T00:48:02.400 INFO:teuthology.orchestra.run.smithi106.stdout: File: /dev/nvme1n1 2024-03-10T00:48:02.400 INFO:teuthology.orchestra.run.smithi106.stdout: Size: 0 Blocks: 0 IO Block: 4096 block special file 2024-03-10T00:48:02.400 INFO:teuthology.orchestra.run.smithi106.stdout:Device: 5h/5d Inode: 762 Links: 1 Device type: 103,2 2024-03-10T00:48:02.401 INFO:teuthology.orchestra.run.smithi106.stdout:Access: (0660/brw-rw----) Uid: ( 0/ root) Gid: ( 6/ disk) 2024-03-10T00:48:02.401 INFO:teuthology.orchestra.run.smithi106.stdout:Context: system_u:object_r:fixed_disk_device_t:s0 2024-03-10T00:48:02.401 INFO:teuthology.orchestra.run.smithi106.stdout:Access: 2024-03-10 00:47:57.223015525 +0000 2024-03-10T00:48:02.401 INFO:teuthology.orchestra.run.smithi106.stdout:Modify: 2024-03-10 00:45:12.945965650 +0000 2024-03-10T00:48:02.401 INFO:teuthology.orchestra.run.smithi106.stdout:Change: 2024-03-10 00:45:12.945965650 +0000 2024-03-10T00:48:02.401 INFO:teuthology.orchestra.run.smithi106.stdout: Birth: 2024-03-10 00:45:12.933965916 +0000 2024-03-10T00:48:02.401 DEBUG:teuthology.orchestra.run.smithi106:> sudo dd if=/dev/nvme1n1 of=/dev/null count=1 2024-03-10T00:48:02.470 INFO:teuthology.orchestra.run.smithi106.stderr:1+0 records in 2024-03-10T00:48:02.470 INFO:teuthology.orchestra.run.smithi106.stderr:1+0 records out 2024-03-10T00:48:02.470 INFO:teuthology.orchestra.run.smithi106.stderr:512 bytes copied, 0.000277984 s, 1.8 MB/s 2024-03-10T00:48:02.471 DEBUG:teuthology.orchestra.run.smithi106:> ! mount | grep -v devtmpfs | grep -q /dev/nvme1n1 2024-03-10T00:48:02.529 DEBUG:teuthology.orchestra.run.smithi106:> stat /dev/nvme2n1 2024-03-10T00:48:02.585 INFO:teuthology.orchestra.run.smithi106.stdout: File: /dev/nvme2n1 2024-03-10T00:48:02.585 INFO:teuthology.orchestra.run.smithi106.stdout: Size: 0 Blocks: 0 IO Block: 4096 block special file 2024-03-10T00:48:02.585 INFO:teuthology.orchestra.run.smithi106.stdout:Device: 5h/5d Inode: 774 Links: 1 Device type: 103,4 2024-03-10T00:48:02.585 INFO:teuthology.orchestra.run.smithi106.stdout:Access: (0660/brw-rw----) Uid: ( 0/ root) Gid: ( 6/ disk) 2024-03-10T00:48:02.586 INFO:teuthology.orchestra.run.smithi106.stdout:Context: system_u:object_r:fixed_disk_device_t:s0 2024-03-10T00:48:02.586 INFO:teuthology.orchestra.run.smithi106.stdout:Access: 2024-03-10 00:47:57.234015285 +0000 2024-03-10T00:48:02.586 INFO:teuthology.orchestra.run.smithi106.stdout:Modify: 2024-03-10 00:45:13.103962148 +0000 2024-03-10T00:48:02.586 INFO:teuthology.orchestra.run.smithi106.stdout:Change: 2024-03-10 00:45:13.103962148 +0000 2024-03-10T00:48:02.586 INFO:teuthology.orchestra.run.smithi106.stdout: Birth: 2024-03-10 00:45:13.091962414 +0000 2024-03-10T00:48:02.586 DEBUG:teuthology.orchestra.run.smithi106:> sudo dd if=/dev/nvme2n1 of=/dev/null count=1 2024-03-10T00:48:02.653 INFO:teuthology.orchestra.run.smithi106.stderr:1+0 records in 2024-03-10T00:48:02.653 INFO:teuthology.orchestra.run.smithi106.stderr:1+0 records out 2024-03-10T00:48:02.653 INFO:teuthology.orchestra.run.smithi106.stderr:512 bytes copied, 0.000252974 s, 2.0 MB/s 2024-03-10T00:48:02.654 DEBUG:teuthology.orchestra.run.smithi106:> ! mount | grep -v devtmpfs | grep -q /dev/nvme2n1 2024-03-10T00:48:02.710 DEBUG:teuthology.orchestra.run.smithi106:> stat /dev/nvme3n1 2024-03-10T00:48:02.765 INFO:teuthology.orchestra.run.smithi106.stdout: File: /dev/nvme3n1 2024-03-10T00:48:02.765 INFO:teuthology.orchestra.run.smithi106.stdout: Size: 0 Blocks: 0 IO Block: 4096 block special file 2024-03-10T00:48:02.766 INFO:teuthology.orchestra.run.smithi106.stdout:Device: 5h/5d Inode: 787 Links: 1 Device type: 103,6 2024-03-10T00:48:02.766 INFO:teuthology.orchestra.run.smithi106.stdout:Access: (0660/brw-rw----) Uid: ( 0/ root) Gid: ( 6/ disk) 2024-03-10T00:48:02.766 INFO:teuthology.orchestra.run.smithi106.stdout:Context: system_u:object_r:fixed_disk_device_t:s0 2024-03-10T00:48:02.766 INFO:teuthology.orchestra.run.smithi106.stdout:Access: 2024-03-10 00:47:57.244015066 +0000 2024-03-10T00:48:02.766 INFO:teuthology.orchestra.run.smithi106.stdout:Modify: 2024-03-10 00:45:13.210959776 +0000 2024-03-10T00:48:02.766 INFO:teuthology.orchestra.run.smithi106.stdout:Change: 2024-03-10 00:45:13.210959776 +0000 2024-03-10T00:48:02.766 INFO:teuthology.orchestra.run.smithi106.stdout: Birth: 2024-03-10 00:45:13.199960020 +0000 2024-03-10T00:48:02.766 DEBUG:teuthology.orchestra.run.smithi106:> sudo dd if=/dev/nvme3n1 of=/dev/null count=1 2024-03-10T00:48:02.833 INFO:teuthology.orchestra.run.smithi106.stderr:1+0 records in 2024-03-10T00:48:02.833 INFO:teuthology.orchestra.run.smithi106.stderr:1+0 records out 2024-03-10T00:48:02.833 INFO:teuthology.orchestra.run.smithi106.stderr:512 bytes copied, 0.000310458 s, 1.6 MB/s 2024-03-10T00:48:02.835 DEBUG:teuthology.orchestra.run.smithi106:> ! mount | grep -v devtmpfs | grep -q /dev/nvme3n1 2024-03-10T00:48:02.890 DEBUG:teuthology.orchestra.run.smithi106:> stat /dev/nvme4n1 2024-03-10T00:48:02.945 INFO:teuthology.orchestra.run.smithi106.stdout: File: /dev/nvme4n1 2024-03-10T00:48:02.945 INFO:teuthology.orchestra.run.smithi106.stdout: Size: 0 Blocks: 0 IO Block: 4096 block special file 2024-03-10T00:48:02.945 INFO:teuthology.orchestra.run.smithi106.stdout:Device: 5h/5d Inode: 800 Links: 1 Device type: 103,8 2024-03-10T00:48:02.945 INFO:teuthology.orchestra.run.smithi106.stdout:Access: (0660/brw-rw----) Uid: ( 0/ root) Gid: ( 6/ disk) 2024-03-10T00:48:02.946 INFO:teuthology.orchestra.run.smithi106.stdout:Context: system_u:object_r:fixed_disk_device_t:s0 2024-03-10T00:48:02.946 INFO:teuthology.orchestra.run.smithi106.stdout:Access: 2024-03-10 00:47:57.255014825 +0000 2024-03-10T00:48:02.946 INFO:teuthology.orchestra.run.smithi106.stdout:Modify: 2024-03-10 00:45:13.367956296 +0000 2024-03-10T00:48:02.946 INFO:teuthology.orchestra.run.smithi106.stdout:Change: 2024-03-10 00:45:13.367956296 +0000 2024-03-10T00:48:02.946 INFO:teuthology.orchestra.run.smithi106.stdout: Birth: 2024-03-10 00:45:13.359956474 +0000 2024-03-10T00:48:02.946 DEBUG:teuthology.orchestra.run.smithi106:> sudo dd if=/dev/nvme4n1 of=/dev/null count=1 2024-03-10T00:48:03.011 INFO:teuthology.orchestra.run.smithi106.stderr:1+0 records in 2024-03-10T00:48:03.011 INFO:teuthology.orchestra.run.smithi106.stderr:1+0 records out 2024-03-10T00:48:03.012 INFO:teuthology.orchestra.run.smithi106.stderr:512 bytes copied, 0.000217032 s, 2.4 MB/s 2024-03-10T00:48:03.013 DEBUG:teuthology.orchestra.run.smithi106:> ! mount | grep -v devtmpfs | grep -q /dev/nvme4n1 2024-03-10T00:48:03.025 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:48:02 smithi170 ceph-mon[27620]: from='client.24109 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "2;smithi106=y;smithi170=x", "target": ["mon-mgr", ""]}]: dispatch 2024-03-10T00:48:03.025 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:48:02 smithi170 ceph-mon[27620]: Saving service mgr spec with placement smithi106=y;smithi170=x;count:2 2024-03-10T00:48:03.025 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:48:02 smithi170 ceph-mon[27620]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:48:03.025 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:48:02 smithi170 ceph-mon[27620]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-10T00:48:03.026 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:48:02 smithi170 ceph-mon[27620]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-10T00:48:03.026 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:48:02 smithi170 ceph-mon[27620]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-10T00:48:03.026 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:48:02 smithi170 ceph-mon[27620]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:48:03.026 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:48:02 smithi170 ceph-mon[27620]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-10T00:48:03.026 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:48:02 smithi170 ceph-mon[27620]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-03-10T00:48:03.027 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:48:02 smithi170 ceph-mon[27620]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-10T00:48:03.027 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:48:02 smithi170 ceph-mon[27620]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-10T00:48:03.027 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:48:02 smithi170 ceph-mon[27620]: Deploying daemon mgr.x on smithi170 2024-03-10T00:48:03.069 DEBUG:teuthology.orchestra.run.smithi170:> set -ex 2024-03-10T00:48:03.069 DEBUG:teuthology.orchestra.run.smithi170:> dd if=/scratch_devs of=/dev/stdout 2024-03-10T00:48:03.085 DEBUG:teuthology.misc:devs=['/dev/nvme1n1', '/dev/nvme2n1', '/dev/nvme3n1', '/dev/nvme4n1'] 2024-03-10T00:48:03.085 DEBUG:teuthology.orchestra.run.smithi170:> stat /dev/nvme1n1 2024-03-10T00:48:03.088 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:48:02 smithi106 ceph-mon[25742]: from='client.24109 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "2;smithi106=y;smithi170=x", "target": ["mon-mgr", ""]}]: dispatch 2024-03-10T00:48:03.089 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:48:02 smithi106 ceph-mon[25742]: Saving service mgr spec with placement smithi106=y;smithi170=x;count:2 2024-03-10T00:48:03.089 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:48:02 smithi106 ceph-mon[25742]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:48:03.089 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:48:02 smithi106 ceph-mon[25742]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-10T00:48:03.089 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:48:02 smithi106 ceph-mon[25742]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-10T00:48:03.089 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:48:02 smithi106 ceph-mon[25742]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-10T00:48:03.089 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:48:02 smithi106 ceph-mon[25742]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:48:03.089 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:48:02 smithi106 ceph-mon[25742]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-10T00:48:03.089 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:48:02 smithi106 ceph-mon[25742]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-03-10T00:48:03.089 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:48:02 smithi106 ceph-mon[25742]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-10T00:48:03.089 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:48:02 smithi106 ceph-mon[25742]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-10T00:48:03.089 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:48:02 smithi106 ceph-mon[25742]: Deploying daemon mgr.x on smithi170 2024-03-10T00:48:03.089 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:48:02 smithi106 ceph-mon[33460]: from='client.24109 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "2;smithi106=y;smithi170=x", "target": ["mon-mgr", ""]}]: dispatch 2024-03-10T00:48:03.090 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:48:02 smithi106 ceph-mon[33460]: Saving service mgr spec with placement smithi106=y;smithi170=x;count:2 2024-03-10T00:48:03.090 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:48:02 smithi106 ceph-mon[33460]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:48:03.090 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:48:02 smithi106 ceph-mon[33460]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-10T00:48:03.090 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:48:02 smithi106 ceph-mon[33460]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-10T00:48:03.090 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:48:02 smithi106 ceph-mon[33460]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-10T00:48:03.090 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:48:02 smithi106 ceph-mon[33460]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:48:03.090 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:48:02 smithi106 ceph-mon[33460]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-10T00:48:03.090 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:48:02 smithi106 ceph-mon[33460]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-03-10T00:48:03.090 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:48:02 smithi106 ceph-mon[33460]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-10T00:48:03.090 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:48:02 smithi106 ceph-mon[33460]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-10T00:48:03.090 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:48:02 smithi106 ceph-mon[33460]: Deploying daemon mgr.x on smithi170 2024-03-10T00:48:03.140 INFO:teuthology.orchestra.run.smithi170.stdout: File: /dev/nvme1n1 2024-03-10T00:48:03.140 INFO:teuthology.orchestra.run.smithi170.stdout: Size: 0 Blocks: 0 IO Block: 4096 block special file 2024-03-10T00:48:03.140 INFO:teuthology.orchestra.run.smithi170.stdout:Device: 5h/5d Inode: 764 Links: 1 Device type: 103,2 2024-03-10T00:48:03.140 INFO:teuthology.orchestra.run.smithi170.stdout:Access: (0660/brw-rw----) Uid: ( 0/ root) Gid: ( 6/ disk) 2024-03-10T00:48:03.140 INFO:teuthology.orchestra.run.smithi170.stdout:Context: system_u:object_r:fixed_disk_device_t:s0 2024-03-10T00:48:03.140 INFO:teuthology.orchestra.run.smithi170.stdout:Access: 2024-03-10 00:47:54.476601737 +0000 2024-03-10T00:48:03.140 INFO:teuthology.orchestra.run.smithi170.stdout:Modify: 2024-03-10 00:45:14.570261006 +0000 2024-03-10T00:48:03.140 INFO:teuthology.orchestra.run.smithi170.stdout:Change: 2024-03-10 00:45:14.570261006 +0000 2024-03-10T00:48:03.140 INFO:teuthology.orchestra.run.smithi170.stdout: Birth: 2024-03-10 00:45:14.563262079 +0000 2024-03-10T00:48:03.141 DEBUG:teuthology.orchestra.run.smithi170:> sudo dd if=/dev/nvme1n1 of=/dev/null count=1 2024-03-10T00:48:03.206 INFO:teuthology.orchestra.run.smithi170.stderr:1+0 records in 2024-03-10T00:48:03.206 INFO:teuthology.orchestra.run.smithi170.stderr:1+0 records out 2024-03-10T00:48:03.206 INFO:teuthology.orchestra.run.smithi170.stderr:512 bytes copied, 0.00012692 s, 4.0 MB/s 2024-03-10T00:48:03.207 DEBUG:teuthology.orchestra.run.smithi170:> ! mount | grep -v devtmpfs | grep -q /dev/nvme1n1 2024-03-10T00:48:03.263 DEBUG:teuthology.orchestra.run.smithi170:> stat /dev/nvme2n1 2024-03-10T00:48:03.318 INFO:teuthology.orchestra.run.smithi170.stdout: File: /dev/nvme2n1 2024-03-10T00:48:03.318 INFO:teuthology.orchestra.run.smithi170.stdout: Size: 0 Blocks: 0 IO Block: 4096 block special file 2024-03-10T00:48:03.318 INFO:teuthology.orchestra.run.smithi170.stdout:Device: 5h/5d Inode: 775 Links: 1 Device type: 103,4 2024-03-10T00:48:03.318 INFO:teuthology.orchestra.run.smithi170.stdout:Access: (0660/brw-rw----) Uid: ( 0/ root) Gid: ( 6/ disk) 2024-03-10T00:48:03.318 INFO:teuthology.orchestra.run.smithi170.stdout:Context: system_u:object_r:fixed_disk_device_t:s0 2024-03-10T00:48:03.318 INFO:teuthology.orchestra.run.smithi170.stdout:Access: 2024-03-10 00:47:54.485601531 +0000 2024-03-10T00:48:03.318 INFO:teuthology.orchestra.run.smithi170.stdout:Modify: 2024-03-10 00:45:14.720252721 +0000 2024-03-10T00:48:03.318 INFO:teuthology.orchestra.run.smithi170.stdout:Change: 2024-03-10 00:45:14.720252721 +0000 2024-03-10T00:48:03.318 INFO:teuthology.orchestra.run.smithi170.stdout: Birth: 2024-03-10 00:45:14.713252881 +0000 2024-03-10T00:48:03.319 DEBUG:teuthology.orchestra.run.smithi170:> sudo dd if=/dev/nvme2n1 of=/dev/null count=1 2024-03-10T00:48:03.384 INFO:teuthology.orchestra.run.smithi170.stderr:1+0 records in 2024-03-10T00:48:03.384 INFO:teuthology.orchestra.run.smithi170.stderr:1+0 records out 2024-03-10T00:48:03.384 INFO:teuthology.orchestra.run.smithi170.stderr:512 bytes copied, 0.000124719 s, 4.1 MB/s 2024-03-10T00:48:03.385 DEBUG:teuthology.orchestra.run.smithi170:> ! mount | grep -v devtmpfs | grep -q /dev/nvme2n1 2024-03-10T00:48:03.440 DEBUG:teuthology.orchestra.run.smithi170:> stat /dev/nvme3n1 2024-03-10T00:48:03.496 INFO:teuthology.orchestra.run.smithi170.stdout: File: /dev/nvme3n1 2024-03-10T00:48:03.497 INFO:teuthology.orchestra.run.smithi170.stdout: Size: 0 Blocks: 0 IO Block: 4096 block special file 2024-03-10T00:48:03.497 INFO:teuthology.orchestra.run.smithi170.stdout:Device: 5h/5d Inode: 787 Links: 1 Device type: 103,6 2024-03-10T00:48:03.497 INFO:teuthology.orchestra.run.smithi170.stdout:Access: (0660/brw-rw----) Uid: ( 0/ root) Gid: ( 6/ disk) 2024-03-10T00:48:03.497 INFO:teuthology.orchestra.run.smithi170.stdout:Context: system_u:object_r:fixed_disk_device_t:s0 2024-03-10T00:48:03.497 INFO:teuthology.orchestra.run.smithi170.stdout:Access: 2024-03-10 00:47:54.494601325 +0000 2024-03-10T00:48:03.497 INFO:teuthology.orchestra.run.smithi170.stdout:Modify: 2024-03-10 00:45:14.874249211 +0000 2024-03-10T00:48:03.497 INFO:teuthology.orchestra.run.smithi170.stdout:Change: 2024-03-10 00:45:14.874249211 +0000 2024-03-10T00:48:03.497 INFO:teuthology.orchestra.run.smithi170.stdout: Birth: 2024-03-10 00:45:14.864249439 +0000 2024-03-10T00:48:03.498 DEBUG:teuthology.orchestra.run.smithi170:> sudo dd if=/dev/nvme3n1 of=/dev/null count=1 2024-03-10T00:48:03.573 INFO:teuthology.orchestra.run.smithi170.stderr:1+0 records in 2024-03-10T00:48:03.573 INFO:teuthology.orchestra.run.smithi170.stderr:1+0 records out 2024-03-10T00:48:03.573 INFO:teuthology.orchestra.run.smithi170.stderr:512 bytes copied, 0.000164113 s, 3.1 MB/s 2024-03-10T00:48:03.574 DEBUG:teuthology.orchestra.run.smithi170:> ! mount | grep -v devtmpfs | grep -q /dev/nvme3n1 2024-03-10T00:48:03.635 DEBUG:teuthology.orchestra.run.smithi170:> stat /dev/nvme4n1 2024-03-10T00:48:03.690 INFO:teuthology.orchestra.run.smithi170.stdout: File: /dev/nvme4n1 2024-03-10T00:48:03.690 INFO:teuthology.orchestra.run.smithi170.stdout: Size: 0 Blocks: 0 IO Block: 4096 block special file 2024-03-10T00:48:03.690 INFO:teuthology.orchestra.run.smithi170.stdout:Device: 5h/5d Inode: 798 Links: 1 Device type: 103,8 2024-03-10T00:48:03.690 INFO:teuthology.orchestra.run.smithi170.stdout:Access: (0660/brw-rw----) Uid: ( 0/ root) Gid: ( 6/ disk) 2024-03-10T00:48:03.691 INFO:teuthology.orchestra.run.smithi170.stdout:Context: system_u:object_r:fixed_disk_device_t:s0 2024-03-10T00:48:03.691 INFO:teuthology.orchestra.run.smithi170.stdout:Access: 2024-03-10 00:47:54.502601142 +0000 2024-03-10T00:48:03.691 INFO:teuthology.orchestra.run.smithi170.stdout:Modify: 2024-03-10 00:45:15.014246019 +0000 2024-03-10T00:48:03.691 INFO:teuthology.orchestra.run.smithi170.stdout:Change: 2024-03-10 00:45:15.014246019 +0000 2024-03-10T00:48:03.691 INFO:teuthology.orchestra.run.smithi170.stdout: Birth: 2024-03-10 00:45:15.011246088 +0000 2024-03-10T00:48:03.691 DEBUG:teuthology.orchestra.run.smithi170:> sudo dd if=/dev/nvme4n1 of=/dev/null count=1 2024-03-10T00:48:03.756 INFO:teuthology.orchestra.run.smithi170.stderr:1+0 records in 2024-03-10T00:48:03.756 INFO:teuthology.orchestra.run.smithi170.stderr:1+0 records out 2024-03-10T00:48:03.756 INFO:teuthology.orchestra.run.smithi170.stderr:512 bytes copied, 0.000118273 s, 4.3 MB/s 2024-03-10T00:48:03.758 DEBUG:teuthology.orchestra.run.smithi170:> ! mount | grep -v devtmpfs | grep -q /dev/nvme4n1 2024-03-10T00:48:03.814 INFO:tasks.cephadm:Deploying osd.0 on smithi106 with /dev/nvme4n1... 2024-03-10T00:48:03.815 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:831bc188fa69f143bbba69ac76119e784500e8e3 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 75c37498-de77-11ee-95c7-87774f69a715 -- ceph-volume lvm zap /dev/nvme4n1 2024-03-10T00:48:03.851 INFO:journalctl@ceph.mgr.x.smithi170.stdout:Mar 10 00:48:03 smithi170 systemd[1]: Starting Ceph mgr.x for 75c37498-de77-11ee-95c7-87774f69a715... 2024-03-10T00:48:04.051 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/75c37498-de77-11ee-95c7-87774f69a715/mon.a/config 2024-03-10T00:48:04.172 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:48:03 smithi106 ceph-mon[25742]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-10T00:48:04.173 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:48:03 smithi106 ceph-mon[33460]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-10T00:48:04.188 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:48:03 smithi170 ceph-mon[27620]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-10T00:48:04.188 INFO:journalctl@ceph.mgr.x.smithi170.stdout:Mar 10 00:48:03 smithi170 podman[30142]: 2024-03-10 00:48:03.979480313 +0000 UTC m=+0.103018575 container create 5fd8a349b80c9a6877bf39c8317aaf7eb359bbe9845db773e80dca74a2b4cfaf (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bc4a24690ce5d7255d490e21058d8c114b55e1b93a3d8345f90b6700ad29dada, name=ceph-75c37498-de77-11ee-95c7-87774f69a715-mgr-x, org.label-schema.build-date=20240222, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-yuri4-testing-2024-03-05-0854-831bc18, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, ceph=True, maintainer=Guillaume Abrioux ) 2024-03-10T00:48:04.188 INFO:journalctl@ceph.mgr.x.smithi170.stdout:Mar 10 00:48:03 smithi170 podman[30142]: 2024-03-10 00:48:03.893555367 +0000 UTC m=+0.017093627 image pull 01a4f6a6bb3e92ab9ab39a2f835684ea9b10e6ec9fb81ac08ebea4165583a538 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bc4a24690ce5d7255d490e21058d8c114b55e1b93a3d8345f90b6700ad29dada 2024-03-10T00:48:04.188 INFO:journalctl@ceph.mgr.x.smithi170.stdout:Mar 10 00:48:04 smithi170 podman[30142]: 2024-03-10 00:48:04.190348171 +0000 UTC m=+0.313886433 container init 5fd8a349b80c9a6877bf39c8317aaf7eb359bbe9845db773e80dca74a2b4cfaf (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bc4a24690ce5d7255d490e21058d8c114b55e1b93a3d8345f90b6700ad29dada, name=ceph-75c37498-de77-11ee-95c7-87774f69a715-mgr-x, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, RELEASE=wip-yuri4-testing-2024-03-05-0854-831bc18, org.label-schema.license=GPLv2, ceph=True, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.build-date=20240222, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5) 2024-03-10T00:48:04.525 INFO:journalctl@ceph.mgr.x.smithi170.stdout:Mar 10 00:48:04 smithi170 podman[30142]: 2024-03-10 00:48:04.196173597 +0000 UTC m=+0.319711859 container start 5fd8a349b80c9a6877bf39c8317aaf7eb359bbe9845db773e80dca74a2b4cfaf (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bc4a24690ce5d7255d490e21058d8c114b55e1b93a3d8345f90b6700ad29dada, name=ceph-75c37498-de77-11ee-95c7-87774f69a715-mgr-x, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, GIT_BRANCH=HEAD, org.label-schema.build-date=20240222, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_CLEAN=True, io.buildah.version=1.33.5, RELEASE=wip-yuri4-testing-2024-03-05-0854-831bc18, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-10T00:48:04.525 INFO:journalctl@ceph.mgr.x.smithi170.stdout:Mar 10 00:48:04 smithi170 ceph-mgr[30180]: -- 172.21.15.170:0/4128709804 <== mon.1 v2:172.21.15.170:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x5587041b45a0 con 0x558704188c00 2024-03-10T00:48:04.525 INFO:journalctl@ceph.mgr.x.smithi170.stdout:Mar 10 00:48:04 smithi170 bash[30142]: 5fd8a349b80c9a6877bf39c8317aaf7eb359bbe9845db773e80dca74a2b4cfaf 2024-03-10T00:48:04.525 INFO:journalctl@ceph.mgr.x.smithi170.stdout:Mar 10 00:48:04 smithi170 systemd[1]: Started Ceph mgr.x for 75c37498-de77-11ee-95c7-87774f69a715. 2024-03-10T00:48:05.025 INFO:journalctl@ceph.mgr.x.smithi170.stdout:Mar 10 00:48:04 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-mgr-x[30157]: 2024-03-10T00:48:04.755+0000 7f8194aa7200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-10T00:48:05.025 INFO:journalctl@ceph.mgr.x.smithi170.stdout:Mar 10 00:48:04 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-mgr-x[30157]: 2024-03-10T00:48:04.847+0000 7f8194aa7200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-10T00:48:05.621 INFO:journalctl@ceph.mgr.x.smithi170.stdout:Mar 10 00:48:05 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-mgr-x[30157]: 2024-03-10T00:48:05.510+0000 7f8194aa7200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-10T00:48:05.621 INFO:journalctl@ceph.mgr.x.smithi170.stdout:Mar 10 00:48:05 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-mgr-x[30157]: 2024-03-10T00:48:05.604+0000 7f8194aa7200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-10T00:48:05.621 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:48:05 smithi170 ceph-mon[27620]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:48:05.622 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:48:05 smithi170 ceph-mon[27620]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:48:05.622 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:48:05 smithi170 ceph-mon[27620]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:48:05.622 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:48:05 smithi170 ceph-mon[27620]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:48:05.622 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:48:05 smithi170 ceph-mon[27620]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-10T00:48:05.622 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:48:05 smithi170 ceph-mon[27620]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-10T00:48:05.622 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:48:05 smithi170 ceph-mon[27620]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-10T00:48:05.622 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:48:05 smithi170 ceph-mon[27620]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:48:05.622 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:48:05 smithi170 ceph-mon[27620]: Metadata not up to date on all hosts. Skipping non agent specs 2024-03-10T00:48:05.622 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:48:05 smithi170 ceph-mon[27620]: Reconfiguring mgr.y (unknown last config time)... 2024-03-10T00:48:05.622 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:48:05 smithi170 ceph-mon[27620]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-10T00:48:05.622 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:48:05 smithi170 ceph-mon[27620]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-10T00:48:05.623 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:48:05 smithi170 ceph-mon[27620]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-10T00:48:05.623 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:48:05 smithi170 ceph-mon[27620]: Reconfiguring daemon mgr.y on smithi106 2024-03-10T00:48:05.623 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:48:05 smithi170 ceph-mon[27620]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-10T00:48:05.628 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:48:05 smithi106 ceph-mon[25742]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:48:05.628 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:48:05 smithi106 ceph-mon[25742]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:48:05.628 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:48:05 smithi106 ceph-mon[25742]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:48:05.628 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:48:05 smithi106 ceph-mon[25742]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:48:05.628 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:48:05 smithi106 ceph-mon[25742]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-10T00:48:05.628 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:48:05 smithi106 ceph-mon[25742]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-10T00:48:05.628 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:48:05 smithi106 ceph-mon[25742]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-10T00:48:05.628 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:48:05 smithi106 ceph-mon[25742]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:48:05.628 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:48:05 smithi106 ceph-mon[25742]: Metadata not up to date on all hosts. Skipping non agent specs 2024-03-10T00:48:05.628 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:48:05 smithi106 ceph-mon[25742]: Reconfiguring mgr.y (unknown last config time)... 2024-03-10T00:48:05.629 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:48:05 smithi106 ceph-mon[25742]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-10T00:48:05.629 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:48:05 smithi106 ceph-mon[25742]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-10T00:48:05.629 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:48:05 smithi106 ceph-mon[25742]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-10T00:48:05.629 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:48:05 smithi106 ceph-mon[25742]: Reconfiguring daemon mgr.y on smithi106 2024-03-10T00:48:05.629 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:48:05 smithi106 ceph-mon[25742]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-10T00:48:05.629 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:48:05 smithi106 ceph-mon[33460]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:48:05.629 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:48:05 smithi106 ceph-mon[33460]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:48:05.629 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:48:05 smithi106 ceph-mon[33460]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:48:05.629 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:48:05 smithi106 ceph-mon[33460]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:48:05.629 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:48:05 smithi106 ceph-mon[33460]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-10T00:48:05.630 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:48:05 smithi106 ceph-mon[33460]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-10T00:48:05.630 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:48:05 smithi106 ceph-mon[33460]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-10T00:48:05.630 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:48:05 smithi106 ceph-mon[33460]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:48:05.630 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:48:05 smithi106 ceph-mon[33460]: Metadata not up to date on all hosts. Skipping non agent specs 2024-03-10T00:48:05.630 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:48:05 smithi106 ceph-mon[33460]: Reconfiguring mgr.y (unknown last config time)... 2024-03-10T00:48:05.630 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:48:05 smithi106 ceph-mon[33460]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-10T00:48:05.630 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:48:05 smithi106 ceph-mon[33460]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-10T00:48:05.630 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:48:05 smithi106 ceph-mon[33460]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-10T00:48:05.630 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:48:05 smithi106 ceph-mon[33460]: Reconfiguring daemon mgr.y on smithi106 2024-03-10T00:48:05.630 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:48:05 smithi106 ceph-mon[33460]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-10T00:48:05.931 INFO:journalctl@ceph.mgr.x.smithi170.stdout:Mar 10 00:48:05 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-mgr-x[30157]: 2024-03-10T00:48:05.754+0000 7f8194aa7200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-10T00:48:05.932 INFO:journalctl@ceph.mgr.x.smithi170.stdout:Mar 10 00:48:05 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-mgr-x[30157]: 2024-03-10T00:48:05.849+0000 7f8194aa7200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-10T00:48:05.932 INFO:journalctl@ceph.mgr.x.smithi170.stdout:Mar 10 00:48:05 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-mgr-x[30157]: 2024-03-10T00:48:05.932+0000 7f8194aa7200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-10T00:48:06.501 INFO:journalctl@ceph.mgr.x.smithi170.stdout:Mar 10 00:48:06 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-mgr-x[30157]: 2024-03-10T00:48:06.204+0000 7f8194aa7200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-10T00:48:06.502 INFO:journalctl@ceph.mgr.x.smithi170.stdout:Mar 10 00:48:06 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-mgr-x[30157]: 2024-03-10T00:48:06.313+0000 7f8194aa7200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-10T00:48:06.502 INFO:journalctl@ceph.mgr.x.smithi170.stdout:Mar 10 00:48:06 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-mgr-x[30157]: 2024-03-10T00:48:06.412+0000 7f8194aa7200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-10T00:48:06.772 INFO:journalctl@ceph.mgr.x.smithi170.stdout:Mar 10 00:48:06 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-mgr-x[30157]: 2024-03-10T00:48:06.510+0000 7f8194aa7200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-10T00:48:06.772 INFO:journalctl@ceph.mgr.x.smithi170.stdout:Mar 10 00:48:06 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-mgr-x[30157]: 2024-03-10T00:48:06.695+0000 7f8194aa7200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-10T00:48:06.914 INFO:teuthology.orchestra.run.smithi106.stderr:--> Zapping: /dev/nvme4n1 2024-03-10T00:48:06.914 INFO:teuthology.orchestra.run.smithi106.stderr:--> --destroy was not specified, but zapping a whole device will remove the partition table 2024-03-10T00:48:06.914 INFO:teuthology.orchestra.run.smithi106.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/nvme4n1 bs=1M count=10 conv=fsync 2024-03-10T00:48:06.914 INFO:teuthology.orchestra.run.smithi106.stderr: stderr: 10+0 records in 2024-03-10T00:48:06.914 INFO:teuthology.orchestra.run.smithi106.stderr:10+0 records out 2024-03-10T00:48:06.914 INFO:teuthology.orchestra.run.smithi106.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0123271 s, 851 MB/s 2024-03-10T00:48:06.914 INFO:teuthology.orchestra.run.smithi106.stderr:--> Zapping successful for: 2024-03-10T00:48:07.025 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:48:06 smithi170 ceph-mon[27620]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:48:07.025 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:48:06 smithi170 ceph-mon[27620]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:48:07.025 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:48:06 smithi170 ceph-mon[27620]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:48:07.025 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:48:06 smithi170 ceph-mon[27620]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-10T00:48:07.025 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:48:06 smithi170 ceph-mon[27620]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-10T00:48:07.025 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:48:06 smithi170 ceph-mon[27620]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-10T00:48:07.025 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:48:06 smithi170 ceph-mon[27620]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:48:07.088 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:48:06 smithi106 ceph-mon[25742]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:48:07.089 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:48:06 smithi106 ceph-mon[25742]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:48:07.089 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:48:06 smithi106 ceph-mon[25742]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:48:07.089 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:48:06 smithi106 ceph-mon[25742]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-10T00:48:07.089 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:48:06 smithi106 ceph-mon[25742]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-10T00:48:07.089 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:48:06 smithi106 ceph-mon[25742]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-10T00:48:07.089 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:48:06 smithi106 ceph-mon[25742]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:48:07.089 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:48:06 smithi106 ceph-mon[33460]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:48:07.089 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:48:06 smithi106 ceph-mon[33460]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:48:07.089 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:48:06 smithi106 ceph-mon[33460]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:48:07.089 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:48:06 smithi106 ceph-mon[33460]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-10T00:48:07.090 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:48:06 smithi106 ceph-mon[33460]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-10T00:48:07.090 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:48:06 smithi106 ceph-mon[33460]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-10T00:48:07.090 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:48:06 smithi106 ceph-mon[33460]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:48:07.476 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:831bc188fa69f143bbba69ac76119e784500e8e3 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 75c37498-de77-11ee-95c7-87774f69a715 -- ceph orch daemon add osd smithi106:/dev/nvme4n1 2024-03-10T00:48:07.708 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/75c37498-de77-11ee-95c7-87774f69a715/mon.a/config 2024-03-10T00:48:07.908 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:48:07 smithi170 ceph-mon[27620]: Metadata not up to date on all hosts. Skipping non agent specs 2024-03-10T00:48:07.908 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:48:07 smithi170 ceph-mon[27620]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-10T00:48:07.908 INFO:journalctl@ceph.mgr.x.smithi170.stdout:Mar 10 00:48:07 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-mgr-x[30157]: 2024-03-10T00:48:07.774+0000 7f8194aa7200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-10T00:48:07.908 INFO:journalctl@ceph.mgr.x.smithi170.stdout:Mar 10 00:48:07 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-mgr-x[30157]: 2024-03-10T00:48:07.909+0000 7f8194aa7200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-10T00:48:08.089 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:48:07 smithi106 ceph-mon[25742]: Metadata not up to date on all hosts. Skipping non agent specs 2024-03-10T00:48:08.089 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:48:07 smithi106 ceph-mon[25742]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-10T00:48:08.089 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:48:07 smithi106 ceph-mon[33460]: Metadata not up to date on all hosts. Skipping non agent specs 2024-03-10T00:48:08.089 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:48:07 smithi106 ceph-mon[33460]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-10T00:48:08.524 INFO:journalctl@ceph.mgr.x.smithi170.stdout:Mar 10 00:48:08 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-mgr-x[30157]: 2024-03-10T00:48:08.251+0000 7f8194aa7200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-10T00:48:08.525 INFO:journalctl@ceph.mgr.x.smithi170.stdout:Mar 10 00:48:08 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-mgr-x[30157]: 2024-03-10T00:48:08.341+0000 7f8194aa7200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-10T00:48:08.989 INFO:journalctl@ceph.mgr.x.smithi170.stdout:Mar 10 00:48:08 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-mgr-x[30157]: 2024-03-10T00:48:08.539+0000 7f8194aa7200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-10T00:48:08.989 INFO:journalctl@ceph.mgr.x.smithi170.stdout:Mar 10 00:48:08 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-mgr-x[30157]: 2024-03-10T00:48:08.716+0000 7f8194aa7200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-10T00:48:09.084 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:48:09 smithi106 ceph-mon[25742]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:48:09.084 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:48:09 smithi106 ceph-mon[25742]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:48:09.084 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:48:09 smithi106 ceph-mon[25742]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-10T00:48:09.085 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:48:09 smithi106 ceph-mon[25742]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:48:09.085 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:48:09 smithi106 ceph-mon[25742]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-10T00:48:09.085 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:48:09 smithi106 ceph-mon[25742]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-10T00:48:09.085 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:48:09 smithi106 ceph-mon[25742]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:48:09.085 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:48:09 smithi106 ceph-mon[25742]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:48:09.085 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:48:09 smithi106 ceph-mon[25742]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:48:09.085 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:48:09 smithi106 ceph-mon[25742]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:48:09.085 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:48:09 smithi106 ceph-mon[25742]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-10T00:48:09.085 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:48:09 smithi106 ceph-mon[33460]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:48:09.085 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:48:09 smithi106 ceph-mon[33460]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:48:09.085 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:48:09 smithi106 ceph-mon[33460]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-10T00:48:09.085 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:48:09 smithi106 ceph-mon[33460]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:48:09.086 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:48:09 smithi106 ceph-mon[33460]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-10T00:48:09.086 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:48:09 smithi106 ceph-mon[33460]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-10T00:48:09.086 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:48:09 smithi106 ceph-mon[33460]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:48:09.086 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:48:09 smithi106 ceph-mon[33460]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:48:09.086 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:48:09 smithi106 ceph-mon[33460]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:48:09.086 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:48:09 smithi106 ceph-mon[33460]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:48:09.086 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:48:09 smithi106 ceph-mon[33460]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-10T00:48:09.249 INFO:journalctl@ceph.mgr.x.smithi170.stdout:Mar 10 00:48:08 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-mgr-x[30157]: 2024-03-10T00:48:08.990+0000 7f8194aa7200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-10T00:48:09.249 INFO:journalctl@ceph.mgr.x.smithi170.stdout:Mar 10 00:48:09 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-mgr-x[30157]: 2024-03-10T00:48:09.249+0000 7f8194aa7200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-10T00:48:09.249 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:48:09 smithi170 ceph-mon[27620]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:48:09.249 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:48:09 smithi170 ceph-mon[27620]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:48:09.249 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:48:09 smithi170 ceph-mon[27620]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-10T00:48:09.249 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:48:09 smithi170 ceph-mon[27620]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:48:09.249 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:48:09 smithi170 ceph-mon[27620]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-10T00:48:09.249 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:48:09 smithi170 ceph-mon[27620]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-10T00:48:09.249 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:48:09 smithi170 ceph-mon[27620]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:48:09.250 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:48:09 smithi170 ceph-mon[27620]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:48:09.250 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:48:09 smithi170 ceph-mon[27620]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:48:09.250 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:48:09 smithi170 ceph-mon[27620]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:48:09.250 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:48:09 smithi170 ceph-mon[27620]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-10T00:48:09.524 INFO:journalctl@ceph.mgr.x.smithi170.stdout:Mar 10 00:48:09 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-mgr-x[30157]: 2024-03-10T00:48:09.484+0000 7f8194aa7200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-10T00:48:10.525 INFO:journalctl@ceph.mgr.x.smithi170.stdout:Mar 10 00:48:10 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-mgr-x[30157]: 2024-03-10T00:48:10.267+0000 7f8194aa7200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-10T00:48:10.525 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:48:10 smithi170 ceph-mon[27620]: from='client.14220 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi106:/dev/nvme4n1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-10T00:48:10.525 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:48:10 smithi170 ceph-mon[27620]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-10T00:48:10.525 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:48:10 smithi170 ceph-mon[27620]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-10T00:48:10.525 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:48:10 smithi170 ceph-mon[27620]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-10T00:48:10.822 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:48:10 smithi106 ceph-mon[33460]: from='client.14220 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi106:/dev/nvme4n1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-10T00:48:10.822 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:48:10 smithi106 ceph-mon[33460]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-10T00:48:10.822 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:48:10 smithi106 ceph-mon[33460]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-10T00:48:10.822 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:48:10 smithi106 ceph-mon[33460]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-10T00:48:10.822 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:48:10 smithi106 ceph-mon[25742]: from='client.14220 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi106:/dev/nvme4n1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-10T00:48:10.822 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:48:10 smithi106 ceph-mon[25742]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-10T00:48:10.822 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:48:10 smithi106 ceph-mon[25742]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-10T00:48:10.822 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:48:10 smithi106 ceph-mon[25742]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-10T00:48:10.956 INFO:journalctl@ceph.mgr.x.smithi170.stdout:Mar 10 00:48:10 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-mgr-x[30157]: 2024-03-10T00:48:10.855+0000 7f8194aa7200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-10T00:48:10.956 INFO:journalctl@ceph.mgr.x.smithi170.stdout:Mar 10 00:48:10 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-mgr-x[30157]: 2024-03-10T00:48:10.957+0000 7f8194aa7200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-10T00:48:11.274 INFO:journalctl@ceph.mgr.x.smithi170.stdout:Mar 10 00:48:11 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-mgr-x[30157]: 2024-03-10T00:48:11.180+0000 7f8194aa7200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-10T00:48:11.775 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:48:11 smithi170 ceph-mon[27620]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-10T00:48:11.775 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:48:11 smithi170 ceph-mon[27620]: Standby manager daemon x started 2024-03-10T00:48:11.775 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:48:11 smithi170 ceph-mon[27620]: from='mgr.? 172.21.15.170:0/2984044372' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-03-10T00:48:11.775 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:48:11 smithi170 ceph-mon[27620]: from='mgr.? 172.21.15.170:0/2984044372' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-10T00:48:11.775 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:48:11 smithi170 ceph-mon[27620]: from='mgr.? 172.21.15.170:0/2984044372' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-03-10T00:48:11.775 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:48:11 smithi170 ceph-mon[27620]: from='mgr.? 172.21.15.170:0/2984044372' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-10T00:48:11.839 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:48:11 smithi106 ceph-mon[25742]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-10T00:48:11.839 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:48:11 smithi106 ceph-mon[25742]: Standby manager daemon x started 2024-03-10T00:48:11.839 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:48:11 smithi106 ceph-mon[25742]: from='mgr.? 172.21.15.170:0/2984044372' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-03-10T00:48:11.839 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:48:11 smithi106 ceph-mon[25742]: from='mgr.? 172.21.15.170:0/2984044372' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-10T00:48:11.839 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:48:11 smithi106 ceph-mon[25742]: from='mgr.? 172.21.15.170:0/2984044372' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-03-10T00:48:11.839 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:48:11 smithi106 ceph-mon[25742]: from='mgr.? 172.21.15.170:0/2984044372' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-10T00:48:11.840 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:48:11 smithi106 ceph-mon[33460]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-10T00:48:11.840 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:48:11 smithi106 ceph-mon[33460]: Standby manager daemon x started 2024-03-10T00:48:11.840 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:48:11 smithi106 ceph-mon[33460]: from='mgr.? 172.21.15.170:0/2984044372' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-03-10T00:48:11.840 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:48:11 smithi106 ceph-mon[33460]: from='mgr.? 172.21.15.170:0/2984044372' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-10T00:48:11.840 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:48:11 smithi106 ceph-mon[33460]: from='mgr.? 172.21.15.170:0/2984044372' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-03-10T00:48:11.840 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:48:11 smithi106 ceph-mon[33460]: from='mgr.? 172.21.15.170:0/2984044372' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-10T00:48:12.775 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:48:12 smithi170 ceph-mon[27620]: mgrmap e14: y(active, since 70s), standbys: x 2024-03-10T00:48:12.775 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:48:12 smithi170 ceph-mon[27620]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-10T00:48:12.775 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:48:12 smithi170 ceph-mon[27620]: from='client.? 172.21.15.106:0/2419594099' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "90836761-e042-415d-b610-3a8383a65d57"}]: dispatch 2024-03-10T00:48:12.775 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:48:12 smithi170 ceph-mon[27620]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "90836761-e042-415d-b610-3a8383a65d57"}]: dispatch 2024-03-10T00:48:12.775 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:48:12 smithi170 ceph-mon[27620]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "90836761-e042-415d-b610-3a8383a65d57"}]': finished 2024-03-10T00:48:12.775 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:48:12 smithi170 ceph-mon[27620]: osdmap e5: 1 total, 0 up, 1 in 2024-03-10T00:48:12.776 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:48:12 smithi170 ceph-mon[27620]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-10T00:48:12.839 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:48:12 smithi106 ceph-mon[25742]: mgrmap e14: y(active, since 70s), standbys: x 2024-03-10T00:48:12.839 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:48:12 smithi106 ceph-mon[25742]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-10T00:48:12.839 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:48:12 smithi106 ceph-mon[25742]: from='client.? 172.21.15.106:0/2419594099' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "90836761-e042-415d-b610-3a8383a65d57"}]: dispatch 2024-03-10T00:48:12.839 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:48:12 smithi106 ceph-mon[25742]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "90836761-e042-415d-b610-3a8383a65d57"}]: dispatch 2024-03-10T00:48:12.839 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:48:12 smithi106 ceph-mon[25742]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "90836761-e042-415d-b610-3a8383a65d57"}]': finished 2024-03-10T00:48:12.839 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:48:12 smithi106 ceph-mon[25742]: osdmap e5: 1 total, 0 up, 1 in 2024-03-10T00:48:12.839 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:48:12 smithi106 ceph-mon[25742]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-10T00:48:12.840 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:48:12 smithi106 ceph-mon[33460]: mgrmap e14: y(active, since 70s), standbys: x 2024-03-10T00:48:12.840 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:48:12 smithi106 ceph-mon[33460]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-10T00:48:12.840 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:48:12 smithi106 ceph-mon[33460]: from='client.? 172.21.15.106:0/2419594099' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "90836761-e042-415d-b610-3a8383a65d57"}]: dispatch 2024-03-10T00:48:12.840 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:48:12 smithi106 ceph-mon[33460]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "90836761-e042-415d-b610-3a8383a65d57"}]: dispatch 2024-03-10T00:48:12.840 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:48:12 smithi106 ceph-mon[33460]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "90836761-e042-415d-b610-3a8383a65d57"}]': finished 2024-03-10T00:48:12.840 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:48:12 smithi106 ceph-mon[33460]: osdmap e5: 1 total, 0 up, 1 in 2024-03-10T00:48:12.840 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:48:12 smithi106 ceph-mon[33460]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-10T00:48:13.774 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:48:13 smithi170 ceph-mon[27620]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-10T00:48:13.838 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:48:13 smithi106 ceph-mon[25742]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-10T00:48:13.839 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:48:13 smithi106 ceph-mon[33460]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-10T00:48:14.774 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:48:14 smithi170 ceph-mon[27620]: from='client.? 172.21.15.106:0/1187221144' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-10T00:48:14.838 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:48:14 smithi106 ceph-mon[25742]: from='client.? 172.21.15.106:0/1187221144' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-10T00:48:14.839 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:48:14 smithi106 ceph-mon[33460]: from='client.? 172.21.15.106:0/1187221144' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-10T00:48:15.581 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:48:15 smithi106 ceph-mon[25742]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-10T00:48:15.581 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:48:15 smithi106 ceph-mon[33460]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-10T00:48:15.775 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:48:15 smithi170 ceph-mon[27620]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-10T00:48:18.089 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:48:17 smithi106 ceph-mon[25742]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-10T00:48:18.090 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:48:17 smithi106 ceph-mon[33460]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-10T00:48:18.275 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:48:17 smithi170 ceph-mon[27620]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-10T00:48:20.025 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:48:19 smithi170 ceph-mon[27620]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:48:20.025 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:48:19 smithi170 ceph-mon[27620]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:48:20.025 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:48:19 smithi170 ceph-mon[27620]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:48:20.025 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:48:19 smithi170 ceph-mon[27620]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-10T00:48:20.089 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:48:19 smithi106 ceph-mon[25742]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:48:20.089 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:48:19 smithi106 ceph-mon[25742]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:48:20.089 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:48:19 smithi106 ceph-mon[25742]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:48:20.090 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:48:19 smithi106 ceph-mon[25742]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-10T00:48:20.090 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:48:19 smithi106 ceph-mon[33460]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:48:20.090 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:48:19 smithi106 ceph-mon[33460]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:48:20.090 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:48:19 smithi106 ceph-mon[33460]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:48:20.090 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:48:19 smithi106 ceph-mon[33460]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-10T00:48:21.024 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:48:20 smithi170 ceph-mon[27620]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-03-10T00:48:21.025 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:48:20 smithi170 ceph-mon[27620]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-10T00:48:21.088 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:48:20 smithi106 ceph-mon[25742]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-03-10T00:48:21.089 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:48:20 smithi106 ceph-mon[25742]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-10T00:48:21.089 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:48:20 smithi106 ceph-mon[33460]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-03-10T00:48:21.089 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:48:20 smithi106 ceph-mon[33460]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-10T00:48:21.839 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:48:21 smithi106 ceph-mon[33460]: Deploying daemon osd.0 on smithi106 2024-03-10T00:48:21.839 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:48:21 smithi106 ceph-mon[33460]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-10T00:48:21.839 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:48:21 smithi106 ceph-mon[25742]: Deploying daemon osd.0 on smithi106 2024-03-10T00:48:21.839 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:48:21 smithi106 ceph-mon[25742]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-10T00:48:22.025 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:48:21 smithi170 ceph-mon[27620]: Deploying daemon osd.0 on smithi106 2024-03-10T00:48:22.025 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:48:21 smithi170 ceph-mon[27620]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-10T00:48:23.950 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:48:23 smithi106 ceph-mon[25742]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-10T00:48:23.950 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:48:23 smithi106 ceph-mon[33460]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-10T00:48:24.025 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:48:23 smithi170 ceph-mon[27620]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-10T00:48:26.006 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:48:25 smithi106 ceph-mon[25742]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-10T00:48:26.007 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:48:25 smithi106 ceph-mon[33460]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-10T00:48:26.024 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:48:25 smithi170 ceph-mon[27620]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-10T00:48:27.339 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:48:27 smithi106 ceph-mon[25742]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:48:27.339 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:48:27 smithi106 ceph-mon[25742]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:48:27.339 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:48:27 smithi106 ceph-mon[25742]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:48:27.339 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:48:27 smithi106 ceph-mon[25742]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:48:27.339 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:48:27 smithi106 ceph-mon[25742]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:48:27.339 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:48:27 smithi106 ceph-mon[25742]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-10T00:48:27.339 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:48:27 smithi106 ceph-mon[25742]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-10T00:48:27.339 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:48:27 smithi106 ceph-mon[25742]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-10T00:48:27.340 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:48:27 smithi106 ceph-mon[25742]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:48:27.340 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:48:27 smithi106 ceph-mon[25742]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-10T00:48:27.340 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:48:27 smithi106 ceph-mon[33460]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:48:27.340 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:48:27 smithi106 ceph-mon[33460]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:48:27.340 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:48:27 smithi106 ceph-mon[33460]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:48:27.340 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:48:27 smithi106 ceph-mon[33460]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:48:27.340 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:48:27 smithi106 ceph-mon[33460]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:48:27.340 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:48:27 smithi106 ceph-mon[33460]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-10T00:48:27.340 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:48:27 smithi106 ceph-mon[33460]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-10T00:48:27.340 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:48:27 smithi106 ceph-mon[33460]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-10T00:48:27.340 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:48:27 smithi106 ceph-mon[33460]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:48:27.340 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:48:27 smithi106 ceph-mon[33460]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-10T00:48:27.409 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:48:27 smithi170 ceph-mon[27620]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:48:27.410 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:48:27 smithi170 ceph-mon[27620]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:48:27.410 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:48:27 smithi170 ceph-mon[27620]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:48:27.410 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:48:27 smithi170 ceph-mon[27620]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:48:27.410 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:48:27 smithi170 ceph-mon[27620]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:48:27.410 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:48:27 smithi170 ceph-mon[27620]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-10T00:48:27.410 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:48:27 smithi170 ceph-mon[27620]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-10T00:48:27.410 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:48:27 smithi170 ceph-mon[27620]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-10T00:48:27.410 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:48:27 smithi170 ceph-mon[27620]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:48:27.410 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:48:27 smithi170 ceph-mon[27620]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-10T00:48:29.221 INFO:teuthology.orchestra.run.smithi106.stdout:Created osd(s) 0 on host 'smithi106' 2024-03-10T00:48:30.022 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:48:29 smithi106 ceph-mon[25742]: pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-10T00:48:30.023 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:48:29 smithi106 ceph-mon[25742]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:48:30.023 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:48:29 smithi106 ceph-mon[25742]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-10T00:48:30.023 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:48:29 smithi106 ceph-mon[25742]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:48:30.023 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:48:29 smithi106 ceph-mon[25742]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-10T00:48:30.023 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:48:29 smithi106 ceph-mon[25742]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:48:30.023 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:48:29 smithi106 ceph-mon[25742]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-10T00:48:30.023 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:48:29 smithi106 ceph-mon[25742]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-10T00:48:30.023 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:48:29 smithi106 ceph-mon[25742]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:48:30.023 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:48:29 smithi106 ceph-mon[25742]: from='osd.0 [v2:172.21.15.106:6802/3397660587,v1:172.21.15.106:6803/3397660587]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-03-10T00:48:30.023 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:48:29 smithi106 ceph-mon[25742]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-03-10T00:48:30.023 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:48:29 smithi106 ceph-mon[33460]: pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-10T00:48:30.024 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:48:29 smithi106 ceph-mon[33460]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:48:30.024 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:48:29 smithi106 ceph-mon[33460]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-10T00:48:30.024 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:48:29 smithi106 ceph-mon[33460]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:48:30.024 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:48:29 smithi106 ceph-mon[33460]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-10T00:48:30.024 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:48:29 smithi106 ceph-mon[33460]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:48:30.024 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:48:29 smithi106 ceph-mon[33460]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-10T00:48:30.024 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:48:29 smithi106 ceph-mon[33460]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-10T00:48:30.024 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:48:29 smithi106 ceph-mon[33460]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:48:30.024 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:48:29 smithi106 ceph-mon[33460]: from='osd.0 [v2:172.21.15.106:6802/3397660587,v1:172.21.15.106:6803/3397660587]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-03-10T00:48:30.024 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:48:29 smithi106 ceph-mon[33460]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-03-10T00:48:30.051 DEBUG:teuthology.orchestra.run.smithi106:osd.0> sudo journalctl -f -n 0 -u ceph-75c37498-de77-11ee-95c7-87774f69a715@osd.0.service 2024-03-10T00:48:30.053 INFO:tasks.cephadm:Deploying osd.1 on smithi106 with /dev/nvme3n1... 2024-03-10T00:48:30.054 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:831bc188fa69f143bbba69ac76119e784500e8e3 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 75c37498-de77-11ee-95c7-87774f69a715 -- ceph-volume lvm zap /dev/nvme3n1 2024-03-10T00:48:30.275 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:48:29 smithi170 ceph-mon[27620]: pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-10T00:48:30.275 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:48:29 smithi170 ceph-mon[27620]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:48:30.275 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:48:29 smithi170 ceph-mon[27620]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-10T00:48:30.275 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:48:29 smithi170 ceph-mon[27620]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:48:30.275 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:48:29 smithi170 ceph-mon[27620]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-10T00:48:30.275 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:48:29 smithi170 ceph-mon[27620]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:48:30.275 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:48:29 smithi170 ceph-mon[27620]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-10T00:48:30.275 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:48:29 smithi170 ceph-mon[27620]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-10T00:48:30.275 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:48:29 smithi170 ceph-mon[27620]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:48:30.275 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:48:29 smithi170 ceph-mon[27620]: from='osd.0 [v2:172.21.15.106:6802/3397660587,v1:172.21.15.106:6803/3397660587]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-03-10T00:48:30.275 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:48:29 smithi170 ceph-mon[27620]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-03-10T00:48:30.329 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/75c37498-de77-11ee-95c7-87774f69a715/mon.a/config 2024-03-10T00:48:31.089 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:48:30 smithi106 ceph-mon[25742]: Adjusting osd_memory_target on smithi106 to 15040M 2024-03-10T00:48:31.089 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:48:30 smithi106 ceph-mon[25742]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-03-10T00:48:31.089 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:48:30 smithi106 ceph-mon[25742]: osdmap e6: 1 total, 0 up, 1 in 2024-03-10T00:48:31.089 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:48:30 smithi106 ceph-mon[25742]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-10T00:48:31.089 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:48:30 smithi106 ceph-mon[25742]: from='osd.0 [v2:172.21.15.106:6802/3397660587,v1:172.21.15.106:6803/3397660587]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi106", "root=default"]}]: dispatch 2024-03-10T00:48:31.089 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:48:30 smithi106 ceph-mon[25742]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi106", "root=default"]}]: dispatch 2024-03-10T00:48:31.090 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:48:30 smithi106 ceph-mon[33460]: Adjusting osd_memory_target on smithi106 to 15040M 2024-03-10T00:48:31.090 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:48:30 smithi106 ceph-mon[33460]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-03-10T00:48:31.090 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:48:30 smithi106 ceph-mon[33460]: osdmap e6: 1 total, 0 up, 1 in 2024-03-10T00:48:31.090 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:48:30 smithi106 ceph-mon[33460]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-10T00:48:31.090 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:48:30 smithi106 ceph-mon[33460]: from='osd.0 [v2:172.21.15.106:6802/3397660587,v1:172.21.15.106:6803/3397660587]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi106", "root=default"]}]: dispatch 2024-03-10T00:48:31.090 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:48:30 smithi106 ceph-mon[33460]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi106", "root=default"]}]: dispatch 2024-03-10T00:48:31.274 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:48:30 smithi170 ceph-mon[27620]: Adjusting osd_memory_target on smithi106 to 15040M 2024-03-10T00:48:31.275 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:48:30 smithi170 ceph-mon[27620]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-03-10T00:48:31.275 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:48:30 smithi170 ceph-mon[27620]: osdmap e6: 1 total, 0 up, 1 in 2024-03-10T00:48:31.275 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:48:30 smithi170 ceph-mon[27620]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-10T00:48:31.275 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:48:30 smithi170 ceph-mon[27620]: from='osd.0 [v2:172.21.15.106:6802/3397660587,v1:172.21.15.106:6803/3397660587]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi106", "root=default"]}]: dispatch 2024-03-10T00:48:31.275 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:48:30 smithi170 ceph-mon[27620]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi106", "root=default"]}]: dispatch 2024-03-10T00:48:32.089 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:48:31 smithi106 ceph-mon[25742]: pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-10T00:48:32.089 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:48:31 smithi106 ceph-mon[25742]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi106", "root=default"]}]': finished 2024-03-10T00:48:32.089 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:48:31 smithi106 ceph-mon[25742]: osdmap e7: 1 total, 0 up, 1 in 2024-03-10T00:48:32.090 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:48:31 smithi106 ceph-mon[25742]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-10T00:48:32.090 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:48:31 smithi106 ceph-mon[25742]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-10T00:48:32.090 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:48:31 smithi106 ceph-mon[25742]: from='osd.0 ' entity='osd.0' 2024-03-10T00:48:32.090 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:48:31 smithi106 ceph-mon[33460]: pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-10T00:48:32.091 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:48:31 smithi106 ceph-mon[33460]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi106", "root=default"]}]': finished 2024-03-10T00:48:32.091 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:48:31 smithi106 ceph-mon[33460]: osdmap e7: 1 total, 0 up, 1 in 2024-03-10T00:48:32.091 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:48:31 smithi106 ceph-mon[33460]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-10T00:48:32.091 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:48:31 smithi106 ceph-mon[33460]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-10T00:48:32.091 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:48:31 smithi106 ceph-mon[33460]: from='osd.0 ' entity='osd.0' 2024-03-10T00:48:32.092 INFO:journalctl@ceph.osd.0.smithi106.stdout:Mar 10 00:48:31 smithi106 ceph-75c37498-de77-11ee-95c7-87774f69a715-osd-0[38522]: 2024-03-10T00:48:31.704+0000 7fc49d573700 -1 osd.0 0 waiting for initial osdmap 2024-03-10T00:48:32.092 INFO:journalctl@ceph.osd.0.smithi106.stdout:Mar 10 00:48:31 smithi106 ceph-75c37498-de77-11ee-95c7-87774f69a715-osd-0[38522]: 2024-03-10T00:48:31.707+0000 7fc49535e700 -1 osd.0 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-10T00:48:32.274 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:48:31 smithi170 ceph-mon[27620]: pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-10T00:48:32.275 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:48:31 smithi170 ceph-mon[27620]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi106", "root=default"]}]': finished 2024-03-10T00:48:32.275 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:48:31 smithi170 ceph-mon[27620]: osdmap e7: 1 total, 0 up, 1 in 2024-03-10T00:48:32.275 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:48:31 smithi170 ceph-mon[27620]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-10T00:48:32.275 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:48:31 smithi170 ceph-mon[27620]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-10T00:48:32.275 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:48:31 smithi170 ceph-mon[27620]: from='osd.0 ' entity='osd.0' 2024-03-10T00:48:32.277 INFO:teuthology.orchestra.run.smithi106.stderr:--> Zapping: /dev/nvme3n1 2024-03-10T00:48:32.277 INFO:teuthology.orchestra.run.smithi106.stderr:--> --destroy was not specified, but zapping a whole device will remove the partition table 2024-03-10T00:48:32.277 INFO:teuthology.orchestra.run.smithi106.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/nvme3n1 bs=1M count=10 conv=fsync 2024-03-10T00:48:32.277 INFO:teuthology.orchestra.run.smithi106.stderr: stderr: 10+0 records in 2024-03-10T00:48:32.277 INFO:teuthology.orchestra.run.smithi106.stderr:10+0 records out 2024-03-10T00:48:32.277 INFO:teuthology.orchestra.run.smithi106.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0130937 s, 801 MB/s 2024-03-10T00:48:32.277 INFO:teuthology.orchestra.run.smithi106.stderr:--> Zapping successful for: 2024-03-10T00:48:32.892 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:48:32 smithi106 ceph-mon[25742]: purged_snaps scrub starts 2024-03-10T00:48:32.892 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:48:32 smithi106 ceph-mon[25742]: purged_snaps scrub ok 2024-03-10T00:48:32.892 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:48:32 smithi106 ceph-mon[25742]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-10T00:48:32.892 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:48:32 smithi106 ceph-mon[25742]: osd.0 [v2:172.21.15.106:6802/3397660587,v1:172.21.15.106:6803/3397660587] boot 2024-03-10T00:48:32.893 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:48:32 smithi106 ceph-mon[25742]: osdmap e8: 1 total, 1 up, 1 in 2024-03-10T00:48:32.893 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:48:32 smithi106 ceph-mon[25742]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-10T00:48:32.893 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:48:32 smithi106 ceph-mon[33460]: purged_snaps scrub starts 2024-03-10T00:48:32.893 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:48:32 smithi106 ceph-mon[33460]: purged_snaps scrub ok 2024-03-10T00:48:32.893 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:48:32 smithi106 ceph-mon[33460]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-10T00:48:32.893 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:48:32 smithi106 ceph-mon[33460]: osd.0 [v2:172.21.15.106:6802/3397660587,v1:172.21.15.106:6803/3397660587] boot 2024-03-10T00:48:32.893 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:48:32 smithi106 ceph-mon[33460]: osdmap e8: 1 total, 1 up, 1 in 2024-03-10T00:48:32.893 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:48:32 smithi106 ceph-mon[33460]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-10T00:48:32.918 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:831bc188fa69f143bbba69ac76119e784500e8e3 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 75c37498-de77-11ee-95c7-87774f69a715 -- ceph orch daemon add osd smithi106:/dev/nvme3n1 2024-03-10T00:48:33.152 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/75c37498-de77-11ee-95c7-87774f69a715/mon.a/config 2024-03-10T00:48:33.274 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:48:32 smithi170 ceph-mon[27620]: purged_snaps scrub starts 2024-03-10T00:48:33.275 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:48:32 smithi170 ceph-mon[27620]: purged_snaps scrub ok 2024-03-10T00:48:33.275 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:48:32 smithi170 ceph-mon[27620]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-10T00:48:33.275 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:48:32 smithi170 ceph-mon[27620]: osd.0 [v2:172.21.15.106:6802/3397660587,v1:172.21.15.106:6803/3397660587] boot 2024-03-10T00:48:33.275 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:48:32 smithi170 ceph-mon[27620]: osdmap e8: 1 total, 1 up, 1 in 2024-03-10T00:48:33.275 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:48:32 smithi170 ceph-mon[27620]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-10T00:48:33.947 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:48:33 smithi106 ceph-mon[25742]: pgmap v44: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-10T00:48:33.948 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:48:33 smithi106 ceph-mon[33460]: pgmap v44: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-10T00:48:34.275 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:48:33 smithi170 ceph-mon[27620]: pgmap v44: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-10T00:48:35.089 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:48:34 smithi106 ceph-mon[25742]: osdmap e9: 1 total, 1 up, 1 in 2024-03-10T00:48:35.089 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:48:34 smithi106 ceph-mon[33460]: osdmap e9: 1 total, 1 up, 1 in 2024-03-10T00:48:35.274 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:48:34 smithi170 ceph-mon[27620]: osdmap e9: 1 total, 1 up, 1 in 2024-03-10T00:48:35.838 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:48:35 smithi106 ceph-mon[25742]: pgmap v46: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-10T00:48:35.839 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:48:35 smithi106 ceph-mon[25742]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-10T00:48:35.839 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:48:35 smithi106 ceph-mon[25742]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-10T00:48:35.839 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:48:35 smithi106 ceph-mon[25742]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-10T00:48:35.839 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:48:35 smithi106 ceph-mon[33460]: pgmap v46: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-10T00:48:35.839 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:48:35 smithi106 ceph-mon[33460]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-10T00:48:35.840 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:48:35 smithi106 ceph-mon[33460]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-10T00:48:35.840 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:48:35 smithi106 ceph-mon[33460]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-10T00:48:36.275 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:48:35 smithi170 ceph-mon[27620]: pgmap v46: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-10T00:48:36.275 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:48:35 smithi170 ceph-mon[27620]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-10T00:48:36.275 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:48:35 smithi170 ceph-mon[27620]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-10T00:48:36.275 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:48:35 smithi170 ceph-mon[27620]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-10T00:48:37.088 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:48:36 smithi106 ceph-mon[25742]: from='client.24149 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi106:/dev/nvme3n1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-10T00:48:37.089 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:48:36 smithi106 ceph-mon[33460]: from='client.24149 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi106:/dev/nvme3n1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-10T00:48:37.274 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:48:36 smithi170 ceph-mon[27620]: from='client.24149 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi106:/dev/nvme3n1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-10T00:48:38.090 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:48:37 smithi106 ceph-mon[25742]: pgmap v47: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-10T00:48:38.091 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:48:37 smithi106 ceph-mon[25742]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:48:38.091 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:48:37 smithi106 ceph-mon[25742]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:48:38.091 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:48:37 smithi106 ceph-mon[25742]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:48:38.091 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:48:37 smithi106 ceph-mon[33460]: pgmap v47: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-10T00:48:38.091 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:48:37 smithi106 ceph-mon[33460]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:48:38.091 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:48:37 smithi106 ceph-mon[33460]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:48:38.091 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:48:37 smithi106 ceph-mon[33460]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:48:38.274 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:48:37 smithi170 ceph-mon[27620]: pgmap v47: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-10T00:48:38.275 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:48:37 smithi170 ceph-mon[27620]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:48:38.275 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:48:37 smithi170 ceph-mon[27620]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:48:38.275 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:48:37 smithi170 ceph-mon[27620]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:48:39.089 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:48:38 smithi106 ceph-mon[25742]: Detected new or changed devices on smithi106 2024-03-10T00:48:39.089 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:48:38 smithi106 ceph-mon[25742]: from='client.? 172.21.15.106:0/1960770035' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "207e1815-38e1-498c-b0cd-602a70cdffa1"}]: dispatch 2024-03-10T00:48:39.089 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:48:38 smithi106 ceph-mon[25742]: from='client.? 172.21.15.106:0/1960770035' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "207e1815-38e1-498c-b0cd-602a70cdffa1"}]': finished 2024-03-10T00:48:39.089 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:48:38 smithi106 ceph-mon[25742]: osdmap e10: 2 total, 1 up, 2 in 2024-03-10T00:48:39.089 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:48:38 smithi106 ceph-mon[25742]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-10T00:48:39.090 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:48:38 smithi106 ceph-mon[33460]: Detected new or changed devices on smithi106 2024-03-10T00:48:39.090 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:48:38 smithi106 ceph-mon[33460]: from='client.? 172.21.15.106:0/1960770035' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "207e1815-38e1-498c-b0cd-602a70cdffa1"}]: dispatch 2024-03-10T00:48:39.090 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:48:38 smithi106 ceph-mon[33460]: from='client.? 172.21.15.106:0/1960770035' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "207e1815-38e1-498c-b0cd-602a70cdffa1"}]': finished 2024-03-10T00:48:39.090 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:48:38 smithi106 ceph-mon[33460]: osdmap e10: 2 total, 1 up, 2 in 2024-03-10T00:48:39.090 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:48:38 smithi106 ceph-mon[33460]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-10T00:48:39.274 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:48:38 smithi170 ceph-mon[27620]: Detected new or changed devices on smithi106 2024-03-10T00:48:39.275 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:48:38 smithi170 ceph-mon[27620]: from='client.? 172.21.15.106:0/1960770035' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "207e1815-38e1-498c-b0cd-602a70cdffa1"}]: dispatch 2024-03-10T00:48:39.275 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:48:38 smithi170 ceph-mon[27620]: from='client.? 172.21.15.106:0/1960770035' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "207e1815-38e1-498c-b0cd-602a70cdffa1"}]': finished 2024-03-10T00:48:39.275 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:48:38 smithi170 ceph-mon[27620]: osdmap e10: 2 total, 1 up, 2 in 2024-03-10T00:48:39.275 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:48:38 smithi170 ceph-mon[27620]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-10T00:48:40.088 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:48:39 smithi106 ceph-mon[25742]: pgmap v49: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-10T00:48:40.089 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:48:39 smithi106 ceph-mon[33460]: pgmap v49: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-10T00:48:40.176 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:48:39 smithi170 ceph-mon[27620]: pgmap v49: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-10T00:48:41.088 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:48:40 smithi106 ceph-mon[25742]: from='client.? 172.21.15.106:0/3573048099' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-10T00:48:41.089 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:48:40 smithi106 ceph-mon[33460]: from='client.? 172.21.15.106:0/3573048099' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-10T00:48:41.274 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:48:40 smithi170 ceph-mon[27620]: from='client.? 172.21.15.106:0/3573048099' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-10T00:48:42.088 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:48:41 smithi106 ceph-mon[25742]: pgmap v50: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-10T00:48:42.089 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:48:41 smithi106 ceph-mon[33460]: pgmap v50: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-10T00:48:42.274 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:48:41 smithi170 ceph-mon[27620]: pgmap v50: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-10T00:48:44.077 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:48:43 smithi106 ceph-mon[25742]: pgmap v51: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-10T00:48:44.126 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:48:43 smithi106 ceph-mon[33460]: pgmap v51: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-10T00:48:44.275 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:48:43 smithi170 ceph-mon[27620]: pgmap v51: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-10T00:48:46.185 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:48:45 smithi106 ceph-mon[25742]: pgmap v52: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-10T00:48:46.186 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:48:45 smithi106 ceph-mon[33460]: pgmap v52: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-10T00:48:46.265 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:48:45 smithi170 ceph-mon[27620]: pgmap v52: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-10T00:48:47.256 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:48:47 smithi170 ceph-mon[27620]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:48:47.257 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:48:47 smithi170 ceph-mon[27620]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:48:47.257 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:48:47 smithi170 ceph-mon[27620]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:48:47.257 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:48:47 smithi170 ceph-mon[27620]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-03-10T00:48:47.257 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:48:47 smithi170 ceph-mon[27620]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-10T00:48:47.257 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:48:47 smithi170 ceph-mon[27620]: Deploying daemon osd.1 on smithi106 2024-03-10T00:48:47.257 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:48:47 smithi170 ceph-mon[27620]: pgmap v53: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-10T00:48:47.339 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:48:47 smithi106 ceph-mon[25742]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:48:47.339 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:48:47 smithi106 ceph-mon[25742]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:48:47.339 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:48:47 smithi106 ceph-mon[25742]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:48:47.339 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:48:47 smithi106 ceph-mon[25742]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-03-10T00:48:47.339 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:48:47 smithi106 ceph-mon[25742]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-10T00:48:47.339 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:48:47 smithi106 ceph-mon[25742]: Deploying daemon osd.1 on smithi106 2024-03-10T00:48:47.339 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:48:47 smithi106 ceph-mon[25742]: pgmap v53: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-10T00:48:47.339 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:48:47 smithi106 ceph-mon[33460]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:48:47.339 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:48:47 smithi106 ceph-mon[33460]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:48:47.340 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:48:47 smithi106 ceph-mon[33460]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:48:47.340 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:48:47 smithi106 ceph-mon[33460]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-03-10T00:48:47.340 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:48:47 smithi106 ceph-mon[33460]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-10T00:48:47.340 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:48:47 smithi106 ceph-mon[33460]: Deploying daemon osd.1 on smithi106 2024-03-10T00:48:47.340 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:48:47 smithi106 ceph-mon[33460]: pgmap v53: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-10T00:48:49.945 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:48:49 smithi106 ceph-mon[25742]: pgmap v54: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-10T00:48:49.945 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:48:49 smithi106 ceph-mon[33460]: pgmap v54: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-10T00:48:50.275 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:48:49 smithi170 ceph-mon[27620]: pgmap v54: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-10T00:48:52.089 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:48:51 smithi106 ceph-mon[25742]: pgmap v55: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-10T00:48:52.089 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:48:51 smithi106 ceph-mon[33460]: pgmap v55: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-10T00:48:52.274 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:48:51 smithi170 ceph-mon[27620]: pgmap v55: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-10T00:48:53.391 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:48:53 smithi106 ceph-mon[25742]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:48:53.391 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:48:53 smithi106 ceph-mon[25742]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:48:53.392 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:48:53 smithi106 ceph-mon[25742]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-10T00:48:53.392 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:48:53 smithi106 ceph-mon[25742]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-10T00:48:53.392 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:48:53 smithi106 ceph-mon[25742]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-10T00:48:53.392 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:48:53 smithi106 ceph-mon[25742]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:48:53.392 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:48:53 smithi106 ceph-mon[25742]: pgmap v56: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-10T00:48:53.392 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:48:53 smithi106 ceph-mon[33460]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:48:53.392 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:48:53 smithi106 ceph-mon[33460]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:48:53.392 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:48:53 smithi106 ceph-mon[33460]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-10T00:48:53.392 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:48:53 smithi106 ceph-mon[33460]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-10T00:48:53.392 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:48:53 smithi106 ceph-mon[33460]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-10T00:48:53.392 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:48:53 smithi106 ceph-mon[33460]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:48:53.393 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:48:53 smithi106 ceph-mon[33460]: pgmap v56: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-10T00:48:53.524 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:48:53 smithi170 ceph-mon[27620]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:48:53.525 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:48:53 smithi170 ceph-mon[27620]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:48:53.525 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:48:53 smithi170 ceph-mon[27620]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-10T00:48:53.525 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:48:53 smithi170 ceph-mon[27620]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-10T00:48:53.525 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:48:53 smithi170 ceph-mon[27620]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-10T00:48:53.525 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:48:53 smithi170 ceph-mon[27620]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:48:53.525 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:48:53 smithi170 ceph-mon[27620]: pgmap v56: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-10T00:48:55.311 INFO:teuthology.orchestra.run.smithi106.stdout:Created osd(s) 1 on host 'smithi106' 2024-03-10T00:48:55.934 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:48:55 smithi106 ceph-mon[25742]: pgmap v57: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-10T00:48:55.935 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:48:55 smithi106 ceph-mon[25742]: from='osd.1 [v2:172.21.15.106:6810/270281423,v1:172.21.15.106:6811/270281423]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-10T00:48:55.935 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:48:55 smithi106 ceph-mon[25742]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:48:55.935 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:48:55 smithi106 ceph-mon[25742]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:48:55.935 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:48:55 smithi106 ceph-mon[25742]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-10T00:48:55.935 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:48:55 smithi106 ceph-mon[25742]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-10T00:48:55.935 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:48:55 smithi106 ceph-mon[25742]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-10T00:48:55.935 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:48:55 smithi106 ceph-mon[25742]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:48:55.935 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:48:55 smithi106 ceph-mon[25742]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-10T00:48:55.935 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:48:55 smithi106 ceph-mon[25742]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-10T00:48:55.935 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:48:55 smithi106 ceph-mon[25742]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:48:55.935 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:48:55 smithi106 ceph-mon[33460]: pgmap v57: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-10T00:48:55.936 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:48:55 smithi106 ceph-mon[33460]: from='osd.1 [v2:172.21.15.106:6810/270281423,v1:172.21.15.106:6811/270281423]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-10T00:48:55.936 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:48:55 smithi106 ceph-mon[33460]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:48:55.936 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:48:55 smithi106 ceph-mon[33460]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:48:55.936 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:48:55 smithi106 ceph-mon[33460]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-10T00:48:55.936 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:48:55 smithi106 ceph-mon[33460]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-10T00:48:55.936 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:48:55 smithi106 ceph-mon[33460]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-10T00:48:55.936 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:48:55 smithi106 ceph-mon[33460]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:48:55.936 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:48:55 smithi106 ceph-mon[33460]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-10T00:48:55.936 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:48:55 smithi106 ceph-mon[33460]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-10T00:48:55.936 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:48:55 smithi106 ceph-mon[33460]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:48:55.961 DEBUG:teuthology.orchestra.run.smithi106:osd.1> sudo journalctl -f -n 0 -u ceph-75c37498-de77-11ee-95c7-87774f69a715@osd.1.service 2024-03-10T00:48:55.964 INFO:tasks.cephadm:Deploying osd.2 on smithi106 with /dev/nvme2n1... 2024-03-10T00:48:55.964 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:831bc188fa69f143bbba69ac76119e784500e8e3 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 75c37498-de77-11ee-95c7-87774f69a715 -- ceph-volume lvm zap /dev/nvme2n1 2024-03-10T00:48:56.235 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/75c37498-de77-11ee-95c7-87774f69a715/mon.a/config 2024-03-10T00:48:56.274 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:48:55 smithi170 ceph-mon[27620]: pgmap v57: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-10T00:48:56.275 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:48:55 smithi170 ceph-mon[27620]: from='osd.1 [v2:172.21.15.106:6810/270281423,v1:172.21.15.106:6811/270281423]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-10T00:48:56.275 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:48:55 smithi170 ceph-mon[27620]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:48:56.275 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:48:55 smithi170 ceph-mon[27620]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:48:56.275 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:48:55 smithi170 ceph-mon[27620]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-10T00:48:56.275 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:48:55 smithi170 ceph-mon[27620]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-10T00:48:56.275 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:48:55 smithi170 ceph-mon[27620]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-10T00:48:56.275 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:48:55 smithi170 ceph-mon[27620]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:48:56.275 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:48:55 smithi170 ceph-mon[27620]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-10T00:48:56.275 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:48:55 smithi170 ceph-mon[27620]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-10T00:48:56.275 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:48:55 smithi170 ceph-mon[27620]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:48:57.255 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:48:56 smithi106 ceph-mon[25742]: Adjusting osd_memory_target on smithi106 to 7520M 2024-03-10T00:48:57.255 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:48:56 smithi106 ceph-mon[25742]: from='osd.1 [v2:172.21.15.106:6810/270281423,v1:172.21.15.106:6811/270281423]' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-03-10T00:48:57.255 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:48:56 smithi106 ceph-mon[25742]: osdmap e11: 2 total, 1 up, 2 in 2024-03-10T00:48:57.255 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:48:56 smithi106 ceph-mon[25742]: from='osd.1 [v2:172.21.15.106:6810/270281423,v1:172.21.15.106:6811/270281423]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi106", "root=default"]}]: dispatch 2024-03-10T00:48:57.255 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:48:56 smithi106 ceph-mon[25742]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-10T00:48:57.255 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:48:56 smithi106 ceph-mon[25742]: from='osd.1 [v2:172.21.15.106:6810/270281423,v1:172.21.15.106:6811/270281423]' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi106", "root=default"]}]': finished 2024-03-10T00:48:57.255 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:48:57 smithi106 ceph-mon[33460]: Adjusting osd_memory_target on smithi106 to 7520M 2024-03-10T00:48:57.256 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:48:57 smithi106 ceph-mon[33460]: from='osd.1 [v2:172.21.15.106:6810/270281423,v1:172.21.15.106:6811/270281423]' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-03-10T00:48:57.256 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:48:57 smithi106 ceph-mon[33460]: osdmap e11: 2 total, 1 up, 2 in 2024-03-10T00:48:57.256 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:48:57 smithi106 ceph-mon[33460]: from='osd.1 [v2:172.21.15.106:6810/270281423,v1:172.21.15.106:6811/270281423]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi106", "root=default"]}]: dispatch 2024-03-10T00:48:57.256 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:48:57 smithi106 ceph-mon[33460]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-10T00:48:57.256 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:48:57 smithi106 ceph-mon[33460]: from='osd.1 [v2:172.21.15.106:6810/270281423,v1:172.21.15.106:6811/270281423]' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi106", "root=default"]}]': finished 2024-03-10T00:48:57.274 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:48:56 smithi170 ceph-mon[27620]: Adjusting osd_memory_target on smithi106 to 7520M 2024-03-10T00:48:57.275 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:48:56 smithi170 ceph-mon[27620]: from='osd.1 [v2:172.21.15.106:6810/270281423,v1:172.21.15.106:6811/270281423]' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-03-10T00:48:57.275 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:48:56 smithi170 ceph-mon[27620]: osdmap e11: 2 total, 1 up, 2 in 2024-03-10T00:48:57.275 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:48:56 smithi170 ceph-mon[27620]: from='osd.1 [v2:172.21.15.106:6810/270281423,v1:172.21.15.106:6811/270281423]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi106", "root=default"]}]: dispatch 2024-03-10T00:48:57.275 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:48:56 smithi170 ceph-mon[27620]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-10T00:48:57.275 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:48:56 smithi170 ceph-mon[27620]: from='osd.1 [v2:172.21.15.106:6810/270281423,v1:172.21.15.106:6811/270281423]' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi106", "root=default"]}]': finished 2024-03-10T00:48:57.510 INFO:journalctl@ceph.osd.1.smithi106.stdout:Mar 10 00:48:57 smithi106 ceph-75c37498-de77-11ee-95c7-87774f69a715-osd-1[42365]: 2024-03-10T00:48:57.255+0000 7f3c2e70d700 -1 osd.1 0 waiting for initial osdmap 2024-03-10T00:48:57.511 INFO:journalctl@ceph.osd.1.smithi106.stdout:Mar 10 00:48:57 smithi106 ceph-75c37498-de77-11ee-95c7-87774f69a715-osd-1[42365]: 2024-03-10T00:48:57.259+0000 7f3c264f8700 -1 osd.1 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-10T00:48:58.089 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:48:57 smithi106 ceph-mon[25742]: pgmap v59: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-10T00:48:58.089 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:48:57 smithi106 ceph-mon[25742]: osdmap e12: 2 total, 1 up, 2 in 2024-03-10T00:48:58.089 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:48:57 smithi106 ceph-mon[25742]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-10T00:48:58.089 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:48:57 smithi106 ceph-mon[25742]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-10T00:48:58.089 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:48:57 smithi106 ceph-mon[25742]: from='osd.1 [v2:172.21.15.106:6810/270281423,v1:172.21.15.106:6811/270281423]' entity='osd.1' 2024-03-10T00:48:58.090 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:48:57 smithi106 ceph-mon[25742]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:48:58.090 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:48:57 smithi106 ceph-mon[25742]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:48:58.090 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:48:57 smithi106 ceph-mon[25742]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-10T00:48:58.090 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:48:57 smithi106 ceph-mon[25742]: Detected new or changed devices on smithi106 2024-03-10T00:48:58.090 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:48:57 smithi106 ceph-mon[25742]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:48:58.090 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:48:57 smithi106 ceph-mon[25742]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-10T00:48:58.091 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:48:57 smithi106 ceph-mon[25742]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-10T00:48:58.091 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:48:57 smithi106 ceph-mon[25742]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-10T00:48:58.091 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:48:57 smithi106 ceph-mon[25742]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:48:58.091 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:48:57 smithi106 ceph-mon[33460]: pgmap v59: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-10T00:48:58.091 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:48:57 smithi106 ceph-mon[33460]: osdmap e12: 2 total, 1 up, 2 in 2024-03-10T00:48:58.091 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:48:57 smithi106 ceph-mon[33460]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-10T00:48:58.091 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:48:57 smithi106 ceph-mon[33460]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-10T00:48:58.092 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:48:57 smithi106 ceph-mon[33460]: from='osd.1 [v2:172.21.15.106:6810/270281423,v1:172.21.15.106:6811/270281423]' entity='osd.1' 2024-03-10T00:48:58.092 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:48:57 smithi106 ceph-mon[33460]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:48:58.092 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:48:57 smithi106 ceph-mon[33460]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:48:58.092 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:48:57 smithi106 ceph-mon[33460]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-10T00:48:58.092 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:48:57 smithi106 ceph-mon[33460]: Detected new or changed devices on smithi106 2024-03-10T00:48:58.092 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:48:57 smithi106 ceph-mon[33460]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:48:58.092 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:48:57 smithi106 ceph-mon[33460]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-10T00:48:58.092 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:48:57 smithi106 ceph-mon[33460]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-10T00:48:58.092 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:48:57 smithi106 ceph-mon[33460]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-10T00:48:58.092 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:48:57 smithi106 ceph-mon[33460]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:48:58.259 INFO:teuthology.orchestra.run.smithi106.stderr:--> Zapping: /dev/nvme2n1 2024-03-10T00:48:58.259 INFO:teuthology.orchestra.run.smithi106.stderr:--> --destroy was not specified, but zapping a whole device will remove the partition table 2024-03-10T00:48:58.260 INFO:teuthology.orchestra.run.smithi106.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/nvme2n1 bs=1M count=10 conv=fsync 2024-03-10T00:48:58.260 INFO:teuthology.orchestra.run.smithi106.stderr: stderr: 10+0 records in 2024-03-10T00:48:58.260 INFO:teuthology.orchestra.run.smithi106.stderr:10+0 records out 2024-03-10T00:48:58.260 INFO:teuthology.orchestra.run.smithi106.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.012951 s, 810 MB/s 2024-03-10T00:48:58.260 INFO:teuthology.orchestra.run.smithi106.stderr:--> Zapping successful for: 2024-03-10T00:48:58.275 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:48:57 smithi170 ceph-mon[27620]: pgmap v59: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-10T00:48:58.275 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:48:57 smithi170 ceph-mon[27620]: osdmap e12: 2 total, 1 up, 2 in 2024-03-10T00:48:58.275 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:48:57 smithi170 ceph-mon[27620]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-10T00:48:58.275 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:48:57 smithi170 ceph-mon[27620]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-10T00:48:58.275 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:48:57 smithi170 ceph-mon[27620]: from='osd.1 [v2:172.21.15.106:6810/270281423,v1:172.21.15.106:6811/270281423]' entity='osd.1' 2024-03-10T00:48:58.275 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:48:57 smithi170 ceph-mon[27620]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:48:58.275 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:48:57 smithi170 ceph-mon[27620]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:48:58.275 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:48:57 smithi170 ceph-mon[27620]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-10T00:48:58.275 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:48:57 smithi170 ceph-mon[27620]: Detected new or changed devices on smithi106 2024-03-10T00:48:58.275 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:48:57 smithi170 ceph-mon[27620]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:48:58.275 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:48:57 smithi170 ceph-mon[27620]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-10T00:48:58.276 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:48:57 smithi170 ceph-mon[27620]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-10T00:48:58.276 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:48:57 smithi170 ceph-mon[27620]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-10T00:48:58.276 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:48:57 smithi170 ceph-mon[27620]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:48:58.826 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:831bc188fa69f143bbba69ac76119e784500e8e3 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 75c37498-de77-11ee-95c7-87774f69a715 -- ceph orch daemon add osd smithi106:/dev/nvme2n1 2024-03-10T00:48:59.088 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:48:58 smithi106 ceph-mon[33460]: purged_snaps scrub starts 2024-03-10T00:48:59.089 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:48:58 smithi106 ceph-mon[33460]: purged_snaps scrub ok 2024-03-10T00:48:59.089 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:48:58 smithi106 ceph-mon[33460]: osd.1 [v2:172.21.15.106:6810/270281423,v1:172.21.15.106:6811/270281423] boot 2024-03-10T00:48:59.089 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:48:58 smithi106 ceph-mon[33460]: osdmap e13: 2 total, 2 up, 2 in 2024-03-10T00:48:59.089 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:48:58 smithi106 ceph-mon[33460]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-10T00:48:59.089 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:48:58 smithi106 ceph-mon[33460]: pgmap v62: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-10T00:48:59.089 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:48:58 smithi106 ceph-mon[25742]: purged_snaps scrub starts 2024-03-10T00:48:59.089 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:48:58 smithi106 ceph-mon[25742]: purged_snaps scrub ok 2024-03-10T00:48:59.089 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:48:58 smithi106 ceph-mon[25742]: osd.1 [v2:172.21.15.106:6810/270281423,v1:172.21.15.106:6811/270281423] boot 2024-03-10T00:48:59.089 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:48:58 smithi106 ceph-mon[25742]: osdmap e13: 2 total, 2 up, 2 in 2024-03-10T00:48:59.090 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:48:58 smithi106 ceph-mon[25742]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-10T00:48:59.090 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:48:58 smithi106 ceph-mon[25742]: pgmap v62: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-10T00:48:59.200 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/75c37498-de77-11ee-95c7-87774f69a715/mon.a/config 2024-03-10T00:48:59.274 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:48:58 smithi170 ceph-mon[27620]: purged_snaps scrub starts 2024-03-10T00:48:59.275 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:48:58 smithi170 ceph-mon[27620]: purged_snaps scrub ok 2024-03-10T00:48:59.275 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:48:58 smithi170 ceph-mon[27620]: osd.1 [v2:172.21.15.106:6810/270281423,v1:172.21.15.106:6811/270281423] boot 2024-03-10T00:48:59.275 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:48:58 smithi170 ceph-mon[27620]: osdmap e13: 2 total, 2 up, 2 in 2024-03-10T00:48:59.275 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:48:58 smithi170 ceph-mon[27620]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-10T00:48:59.275 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:48:58 smithi170 ceph-mon[27620]: pgmap v62: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-10T00:49:00.339 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:49:00 smithi106 ceph-mon[33460]: osdmap e14: 2 total, 2 up, 2 in 2024-03-10T00:49:00.339 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:49:00 smithi106 ceph-mon[25742]: osdmap e14: 2 total, 2 up, 2 in 2024-03-10T00:49:00.524 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:49:00 smithi170 ceph-mon[27620]: osdmap e14: 2 total, 2 up, 2 in 2024-03-10T00:49:01.524 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:49:01 smithi170 ceph-mon[27620]: pgmap v64: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-10T00:49:01.525 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:49:01 smithi170 ceph-mon[27620]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:49:01.525 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:49:01 smithi170 ceph-mon[27620]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:49:01.525 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:49:01 smithi170 ceph-mon[27620]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:49:01.525 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:49:01 smithi170 ceph-mon[27620]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-10T00:49:01.525 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:49:01 smithi170 ceph-mon[27620]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-10T00:49:01.525 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:49:01 smithi170 ceph-mon[27620]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-10T00:49:01.525 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:49:01 smithi170 ceph-mon[27620]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:49:01.539 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:49:01 smithi106 ceph-mon[25742]: pgmap v64: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-10T00:49:01.539 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:49:01 smithi106 ceph-mon[25742]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:49:01.539 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:49:01 smithi106 ceph-mon[25742]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:49:01.539 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:49:01 smithi106 ceph-mon[25742]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:49:01.539 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:49:01 smithi106 ceph-mon[25742]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-10T00:49:01.539 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:49:01 smithi106 ceph-mon[25742]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-10T00:49:01.539 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:49:01 smithi106 ceph-mon[25742]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-10T00:49:01.540 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:49:01 smithi106 ceph-mon[25742]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:49:01.540 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:49:01 smithi106 ceph-mon[33460]: pgmap v64: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-10T00:49:01.540 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:49:01 smithi106 ceph-mon[33460]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:49:01.540 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:49:01 smithi106 ceph-mon[33460]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:49:01.540 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:49:01 smithi106 ceph-mon[33460]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:49:01.540 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:49:01 smithi106 ceph-mon[33460]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-10T00:49:01.540 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:49:01 smithi106 ceph-mon[33460]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-10T00:49:01.540 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:49:01 smithi106 ceph-mon[33460]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-10T00:49:01.540 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:49:01 smithi106 ceph-mon[33460]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:49:02.775 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:49:02 smithi170 ceph-mon[27620]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-10T00:49:02.775 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:49:02 smithi170 ceph-mon[27620]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-10T00:49:02.775 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:49:02 smithi170 ceph-mon[27620]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-10T00:49:02.838 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:49:02 smithi106 ceph-mon[25742]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-10T00:49:02.838 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:49:02 smithi106 ceph-mon[25742]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-10T00:49:02.839 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:49:02 smithi106 ceph-mon[25742]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-10T00:49:02.839 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:49:02 smithi106 ceph-mon[33460]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-10T00:49:02.839 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:49:02 smithi106 ceph-mon[33460]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-10T00:49:02.839 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:49:02 smithi106 ceph-mon[33460]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-10T00:49:03.757 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:49:03 smithi106 ceph-mon[25742]: from='client.14274 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi106:/dev/nvme2n1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-10T00:49:03.758 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:49:03 smithi106 ceph-mon[25742]: pgmap v65: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-10T00:49:03.758 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:49:03 smithi106 ceph-mon[33460]: from='client.14274 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi106:/dev/nvme2n1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-10T00:49:03.758 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:49:03 smithi106 ceph-mon[33460]: pgmap v65: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-10T00:49:03.774 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:49:03 smithi170 ceph-mon[27620]: from='client.14274 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi106:/dev/nvme2n1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-10T00:49:03.774 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:49:03 smithi170 ceph-mon[27620]: pgmap v65: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-10T00:49:06.089 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:49:05 smithi106 ceph-mon[25742]: pgmap v66: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-10T00:49:06.089 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:49:05 smithi106 ceph-mon[25742]: from='client.? 172.21.15.106:0/2698540600' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b8113c87-8517-4b71-b6b4-ad7f2fce167b"}]: dispatch 2024-03-10T00:49:06.089 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:49:05 smithi106 ceph-mon[25742]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b8113c87-8517-4b71-b6b4-ad7f2fce167b"}]: dispatch 2024-03-10T00:49:06.089 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:49:05 smithi106 ceph-mon[25742]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "b8113c87-8517-4b71-b6b4-ad7f2fce167b"}]': finished 2024-03-10T00:49:06.089 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:49:05 smithi106 ceph-mon[25742]: osdmap e15: 3 total, 2 up, 3 in 2024-03-10T00:49:06.090 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:49:05 smithi106 ceph-mon[25742]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-10T00:49:06.090 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:49:05 smithi106 ceph-mon[33460]: pgmap v66: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-10T00:49:06.090 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:49:05 smithi106 ceph-mon[33460]: from='client.? 172.21.15.106:0/2698540600' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b8113c87-8517-4b71-b6b4-ad7f2fce167b"}]: dispatch 2024-03-10T00:49:06.090 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:49:05 smithi106 ceph-mon[33460]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b8113c87-8517-4b71-b6b4-ad7f2fce167b"}]: dispatch 2024-03-10T00:49:06.090 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:49:05 smithi106 ceph-mon[33460]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "b8113c87-8517-4b71-b6b4-ad7f2fce167b"}]': finished 2024-03-10T00:49:06.090 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:49:05 smithi106 ceph-mon[33460]: osdmap e15: 3 total, 2 up, 3 in 2024-03-10T00:49:06.090 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:49:05 smithi106 ceph-mon[33460]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-10T00:49:06.275 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:49:05 smithi170 ceph-mon[27620]: pgmap v66: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-10T00:49:06.275 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:49:05 smithi170 ceph-mon[27620]: from='client.? 172.21.15.106:0/2698540600' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b8113c87-8517-4b71-b6b4-ad7f2fce167b"}]: dispatch 2024-03-10T00:49:06.275 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:49:05 smithi170 ceph-mon[27620]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b8113c87-8517-4b71-b6b4-ad7f2fce167b"}]: dispatch 2024-03-10T00:49:06.275 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:49:05 smithi170 ceph-mon[27620]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "b8113c87-8517-4b71-b6b4-ad7f2fce167b"}]': finished 2024-03-10T00:49:06.275 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:49:05 smithi170 ceph-mon[27620]: osdmap e15: 3 total, 2 up, 3 in 2024-03-10T00:49:06.275 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:49:05 smithi170 ceph-mon[27620]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-10T00:49:07.708 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:49:07 smithi170 ceph-mon[27620]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:49:07.709 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:49:07 smithi170 ceph-mon[27620]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:49:07.709 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:49:07 smithi170 ceph-mon[27620]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:49:07.709 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:49:07 smithi170 ceph-mon[27620]: pgmap v68: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-10T00:49:07.709 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:49:07 smithi170 ceph-mon[27620]: from='client.? 172.21.15.106:0/2453144164' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-10T00:49:07.838 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:49:07 smithi106 ceph-mon[25742]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:49:07.839 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:49:07 smithi106 ceph-mon[25742]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:49:07.839 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:49:07 smithi106 ceph-mon[25742]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:49:07.839 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:49:07 smithi106 ceph-mon[25742]: pgmap v68: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-10T00:49:07.839 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:49:07 smithi106 ceph-mon[25742]: from='client.? 172.21.15.106:0/2453144164' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-10T00:49:07.839 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:49:07 smithi106 ceph-mon[33460]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:49:07.839 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:49:07 smithi106 ceph-mon[33460]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:49:07.839 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:49:07 smithi106 ceph-mon[33460]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:49:07.839 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:49:07 smithi106 ceph-mon[33460]: pgmap v68: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-10T00:49:07.839 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:49:07 smithi106 ceph-mon[33460]: from='client.? 172.21.15.106:0/2453144164' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-10T00:49:09.920 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:49:09 smithi170 ceph-mon[27620]: pgmap v69: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-10T00:49:10.088 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:49:09 smithi106 ceph-mon[25742]: pgmap v69: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-10T00:49:10.089 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:49:09 smithi106 ceph-mon[33460]: pgmap v69: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-10T00:49:12.024 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:49:11 smithi170 ceph-mon[27620]: pgmap v70: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-10T00:49:12.087 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:49:11 smithi106 ceph-mon[25742]: pgmap v70: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-10T00:49:12.087 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:49:11 smithi106 ceph-mon[33460]: pgmap v70: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-10T00:49:13.088 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:49:12 smithi106 ceph-mon[25742]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-03-10T00:49:13.089 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:49:12 smithi106 ceph-mon[25742]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-10T00:49:13.089 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:49:12 smithi106 ceph-mon[33460]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-03-10T00:49:13.089 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:49:12 smithi106 ceph-mon[33460]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-10T00:49:13.274 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:49:12 smithi170 ceph-mon[27620]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-03-10T00:49:13.275 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:49:12 smithi170 ceph-mon[27620]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-10T00:49:14.005 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:49:13 smithi106 ceph-mon[25742]: Deploying daemon osd.2 on smithi106 2024-03-10T00:49:14.006 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:49:13 smithi106 ceph-mon[25742]: pgmap v71: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-10T00:49:14.006 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:49:13 smithi106 ceph-mon[33460]: Deploying daemon osd.2 on smithi106 2024-03-10T00:49:14.006 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:49:13 smithi106 ceph-mon[33460]: pgmap v71: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-10T00:49:14.274 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:49:13 smithi170 ceph-mon[27620]: Deploying daemon osd.2 on smithi106 2024-03-10T00:49:14.275 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:49:13 smithi170 ceph-mon[27620]: pgmap v71: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-10T00:49:16.088 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:49:15 smithi106 ceph-mon[25742]: pgmap v72: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-10T00:49:16.089 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:49:15 smithi106 ceph-mon[33460]: pgmap v72: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-10T00:49:16.274 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:49:15 smithi170 ceph-mon[27620]: pgmap v72: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-10T00:49:18.088 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:49:17 smithi106 ceph-mon[25742]: pgmap v73: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-10T00:49:18.089 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:49:17 smithi106 ceph-mon[33460]: pgmap v73: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-10T00:49:18.274 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:49:17 smithi170 ceph-mon[27620]: pgmap v73: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-10T00:49:19.729 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:49:19 smithi106 ceph-mon[25742]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:49:19.729 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:49:19 smithi106 ceph-mon[25742]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:49:19.729 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:49:19 smithi106 ceph-mon[25742]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-10T00:49:19.729 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:49:19 smithi106 ceph-mon[25742]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-10T00:49:19.729 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:49:19 smithi106 ceph-mon[25742]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-10T00:49:19.729 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:49:19 smithi106 ceph-mon[25742]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:49:19.729 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:49:19 smithi106 ceph-mon[25742]: pgmap v74: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-10T00:49:19.729 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:49:19 smithi106 ceph-mon[33460]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:49:19.730 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:49:19 smithi106 ceph-mon[33460]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:49:19.730 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:49:19 smithi106 ceph-mon[33460]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-10T00:49:19.730 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:49:19 smithi106 ceph-mon[33460]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-10T00:49:19.730 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:49:19 smithi106 ceph-mon[33460]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-10T00:49:19.730 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:49:19 smithi106 ceph-mon[33460]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:49:19.730 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:49:19 smithi106 ceph-mon[33460]: pgmap v74: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-10T00:49:19.774 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:49:19 smithi170 ceph-mon[27620]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:49:19.775 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:49:19 smithi170 ceph-mon[27620]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:49:19.775 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:49:19 smithi170 ceph-mon[27620]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-10T00:49:19.775 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:49:19 smithi170 ceph-mon[27620]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-10T00:49:19.775 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:49:19 smithi170 ceph-mon[27620]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-10T00:49:19.775 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:49:19 smithi170 ceph-mon[27620]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:49:19.775 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:49:19 smithi170 ceph-mon[27620]: pgmap v74: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-10T00:49:22.067 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:49:21 smithi106 ceph-mon[25742]: pgmap v75: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-10T00:49:22.067 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:49:21 smithi106 ceph-mon[25742]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:49:22.067 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:49:21 smithi106 ceph-mon[25742]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:49:22.067 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:49:21 smithi106 ceph-mon[25742]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:49:22.068 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:49:21 smithi106 ceph-mon[25742]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-10T00:49:22.068 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:49:21 smithi106 ceph-mon[25742]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-10T00:49:22.068 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:49:21 smithi106 ceph-mon[25742]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-10T00:49:22.068 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:49:21 smithi106 ceph-mon[25742]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:49:22.068 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:49:21 smithi106 ceph-mon[25742]: from='osd.2 [v2:172.21.15.106:6818/3393056025,v1:172.21.15.106:6819/3393056025]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-10T00:49:22.068 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:49:21 smithi106 ceph-mon[25742]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-10T00:49:22.068 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:49:21 smithi106 ceph-mon[33460]: pgmap v75: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-10T00:49:22.068 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:49:21 smithi106 ceph-mon[33460]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:49:22.068 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:49:21 smithi106 ceph-mon[33460]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:49:22.068 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:49:21 smithi106 ceph-mon[33460]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:49:22.068 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:49:21 smithi106 ceph-mon[33460]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-10T00:49:22.068 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:49:21 smithi106 ceph-mon[33460]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-10T00:49:22.069 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:49:21 smithi106 ceph-mon[33460]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-10T00:49:22.069 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:49:21 smithi106 ceph-mon[33460]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:49:22.069 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:49:21 smithi106 ceph-mon[33460]: from='osd.2 [v2:172.21.15.106:6818/3393056025,v1:172.21.15.106:6819/3393056025]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-10T00:49:22.069 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:49:21 smithi106 ceph-mon[33460]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-10T00:49:22.274 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:49:21 smithi170 ceph-mon[27620]: pgmap v75: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-10T00:49:22.275 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:49:21 smithi170 ceph-mon[27620]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:49:22.275 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:49:21 smithi170 ceph-mon[27620]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:49:22.275 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:49:21 smithi170 ceph-mon[27620]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:49:22.275 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:49:21 smithi170 ceph-mon[27620]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-10T00:49:22.275 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:49:21 smithi170 ceph-mon[27620]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-10T00:49:22.275 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:49:21 smithi170 ceph-mon[27620]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-10T00:49:22.275 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:49:21 smithi170 ceph-mon[27620]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:49:22.275 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:49:21 smithi170 ceph-mon[27620]: from='osd.2 [v2:172.21.15.106:6818/3393056025,v1:172.21.15.106:6819/3393056025]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-10T00:49:22.275 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:49:21 smithi170 ceph-mon[27620]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-10T00:49:23.589 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:49:23 smithi106 ceph-mon[25742]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-03-10T00:49:23.589 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:49:23 smithi106 ceph-mon[25742]: osdmap e16: 3 total, 2 up, 3 in 2024-03-10T00:49:23.589 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:49:23 smithi106 ceph-mon[25742]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-10T00:49:23.589 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:49:23 smithi106 ceph-mon[25742]: from='osd.2 [v2:172.21.15.106:6818/3393056025,v1:172.21.15.106:6819/3393056025]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi106", "root=default"]}]: dispatch 2024-03-10T00:49:23.589 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:49:23 smithi106 ceph-mon[25742]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi106", "root=default"]}]: dispatch 2024-03-10T00:49:23.589 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:49:23 smithi106 ceph-mon[25742]: pgmap v77: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-10T00:49:23.590 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:49:23 smithi106 ceph-mon[33460]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-03-10T00:49:23.590 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:49:23 smithi106 ceph-mon[33460]: osdmap e16: 3 total, 2 up, 3 in 2024-03-10T00:49:23.590 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:49:23 smithi106 ceph-mon[33460]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-10T00:49:23.590 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:49:23 smithi106 ceph-mon[33460]: from='osd.2 [v2:172.21.15.106:6818/3393056025,v1:172.21.15.106:6819/3393056025]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi106", "root=default"]}]: dispatch 2024-03-10T00:49:23.590 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:49:23 smithi106 ceph-mon[33460]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi106", "root=default"]}]: dispatch 2024-03-10T00:49:23.590 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:49:23 smithi106 ceph-mon[33460]: pgmap v77: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-10T00:49:23.774 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:49:23 smithi170 ceph-mon[27620]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-03-10T00:49:23.775 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:49:23 smithi170 ceph-mon[27620]: osdmap e16: 3 total, 2 up, 3 in 2024-03-10T00:49:23.775 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:49:23 smithi170 ceph-mon[27620]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-10T00:49:23.775 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:49:23 smithi170 ceph-mon[27620]: from='osd.2 [v2:172.21.15.106:6818/3393056025,v1:172.21.15.106:6819/3393056025]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi106", "root=default"]}]: dispatch 2024-03-10T00:49:23.775 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:49:23 smithi170 ceph-mon[27620]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi106", "root=default"]}]: dispatch 2024-03-10T00:49:23.775 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:49:23 smithi170 ceph-mon[27620]: pgmap v77: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-10T00:49:23.866 INFO:teuthology.orchestra.run.smithi106.stdout:Created osd(s) 2 on host 'smithi106' 2024-03-10T00:49:24.479 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:49:24 smithi106 ceph-mon[25742]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi106", "root=default"]}]': finished 2024-03-10T00:49:24.479 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:49:24 smithi106 ceph-mon[25742]: osdmap e17: 3 total, 2 up, 3 in 2024-03-10T00:49:24.479 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:49:24 smithi106 ceph-mon[25742]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-10T00:49:24.480 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:49:24 smithi106 ceph-mon[25742]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-10T00:49:24.480 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:49:24 smithi106 ceph-mon[25742]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:49:24.480 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:49:24 smithi106 ceph-mon[25742]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-10T00:49:24.480 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:49:24 smithi106 ceph-mon[25742]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:49:24.480 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:49:24 smithi106 ceph-mon[25742]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-10T00:49:24.480 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:49:24 smithi106 ceph-mon[25742]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-10T00:49:24.480 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:49:24 smithi106 ceph-mon[25742]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:49:24.480 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:49:24 smithi106 ceph-mon[25742]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-10T00:49:24.480 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:49:24 smithi106 ceph-mon[33460]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi106", "root=default"]}]': finished 2024-03-10T00:49:24.480 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:49:24 smithi106 ceph-mon[33460]: osdmap e17: 3 total, 2 up, 3 in 2024-03-10T00:49:24.481 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:49:24 smithi106 ceph-mon[33460]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-10T00:49:24.481 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:49:24 smithi106 ceph-mon[33460]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-10T00:49:24.481 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:49:24 smithi106 ceph-mon[33460]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:49:24.481 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:49:24 smithi106 ceph-mon[33460]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-10T00:49:24.481 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:49:24 smithi106 ceph-mon[33460]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:49:24.481 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:49:24 smithi106 ceph-mon[33460]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-10T00:49:24.481 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:49:24 smithi106 ceph-mon[33460]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-10T00:49:24.481 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:49:24 smithi106 ceph-mon[33460]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:49:24.481 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:49:24 smithi106 ceph-mon[33460]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-10T00:49:24.508 DEBUG:teuthology.orchestra.run.smithi106:osd.2> sudo journalctl -f -n 0 -u ceph-75c37498-de77-11ee-95c7-87774f69a715@osd.2.service 2024-03-10T00:49:24.510 INFO:tasks.cephadm:Deploying osd.3 on smithi106 with /dev/nvme1n1... 2024-03-10T00:49:24.511 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:831bc188fa69f143bbba69ac76119e784500e8e3 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 75c37498-de77-11ee-95c7-87774f69a715 -- ceph-volume lvm zap /dev/nvme1n1 2024-03-10T00:49:24.774 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:49:24 smithi170 ceph-mon[27620]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi106", "root=default"]}]': finished 2024-03-10T00:49:24.775 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:49:24 smithi170 ceph-mon[27620]: osdmap e17: 3 total, 2 up, 3 in 2024-03-10T00:49:24.775 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:49:24 smithi170 ceph-mon[27620]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-10T00:49:24.775 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:49:24 smithi170 ceph-mon[27620]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-10T00:49:24.775 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:49:24 smithi170 ceph-mon[27620]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:49:24.775 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:49:24 smithi170 ceph-mon[27620]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-10T00:49:24.775 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:49:24 smithi170 ceph-mon[27620]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:49:24.775 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:49:24 smithi170 ceph-mon[27620]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-10T00:49:24.775 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:49:24 smithi170 ceph-mon[27620]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-10T00:49:24.775 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:49:24 smithi170 ceph-mon[27620]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:49:24.775 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:49:24 smithi170 ceph-mon[27620]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-10T00:49:24.804 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/75c37498-de77-11ee-95c7-87774f69a715/mon.a/config 2024-03-10T00:49:25.589 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:49:25 smithi106 ceph-mon[25742]: purged_snaps scrub starts 2024-03-10T00:49:25.589 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:49:25 smithi106 ceph-mon[25742]: purged_snaps scrub ok 2024-03-10T00:49:25.589 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:49:25 smithi106 ceph-mon[25742]: osd.2 [v2:172.21.15.106:6818/3393056025,v1:172.21.15.106:6819/3393056025] boot 2024-03-10T00:49:25.589 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:49:25 smithi106 ceph-mon[25742]: osdmap e18: 3 total, 3 up, 3 in 2024-03-10T00:49:25.589 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:49:25 smithi106 ceph-mon[25742]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-10T00:49:25.589 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:49:25 smithi106 ceph-mon[25742]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:49:25.589 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:49:25 smithi106 ceph-mon[25742]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:49:25.589 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:49:25 smithi106 ceph-mon[25742]: Detected new or changed devices on smithi106 2024-03-10T00:49:25.589 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:49:25 smithi106 ceph-mon[25742]: pgmap v80: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-10T00:49:25.589 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:49:25 smithi106 ceph-mon[25742]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:49:25.590 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:49:25 smithi106 ceph-mon[25742]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-10T00:49:25.590 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:49:25 smithi106 ceph-mon[25742]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-10T00:49:25.590 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:49:25 smithi106 ceph-mon[25742]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-10T00:49:25.590 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:49:25 smithi106 ceph-mon[25742]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:49:25.590 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:49:25 smithi106 ceph-mon[25742]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-03-10T00:49:25.590 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:49:25 smithi106 ceph-mon[33460]: purged_snaps scrub starts 2024-03-10T00:49:25.590 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:49:25 smithi106 ceph-mon[33460]: purged_snaps scrub ok 2024-03-10T00:49:25.590 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:49:25 smithi106 ceph-mon[33460]: osd.2 [v2:172.21.15.106:6818/3393056025,v1:172.21.15.106:6819/3393056025] boot 2024-03-10T00:49:25.590 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:49:25 smithi106 ceph-mon[33460]: osdmap e18: 3 total, 3 up, 3 in 2024-03-10T00:49:25.590 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:49:25 smithi106 ceph-mon[33460]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-10T00:49:25.590 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:49:25 smithi106 ceph-mon[33460]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:49:25.591 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:49:25 smithi106 ceph-mon[33460]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:49:25.591 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:49:25 smithi106 ceph-mon[33460]: Detected new or changed devices on smithi106 2024-03-10T00:49:25.591 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:49:25 smithi106 ceph-mon[33460]: pgmap v80: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-10T00:49:25.591 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:49:25 smithi106 ceph-mon[33460]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:49:25.591 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:49:25 smithi106 ceph-mon[33460]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-10T00:49:25.591 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:49:25 smithi106 ceph-mon[33460]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-10T00:49:25.591 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:49:25 smithi106 ceph-mon[33460]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-10T00:49:25.591 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:49:25 smithi106 ceph-mon[33460]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:49:25.591 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:49:25 smithi106 ceph-mon[33460]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-03-10T00:49:25.774 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:49:25 smithi170 ceph-mon[27620]: purged_snaps scrub starts 2024-03-10T00:49:25.775 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:49:25 smithi170 ceph-mon[27620]: purged_snaps scrub ok 2024-03-10T00:49:25.775 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:49:25 smithi170 ceph-mon[27620]: osd.2 [v2:172.21.15.106:6818/3393056025,v1:172.21.15.106:6819/3393056025] boot 2024-03-10T00:49:25.775 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:49:25 smithi170 ceph-mon[27620]: osdmap e18: 3 total, 3 up, 3 in 2024-03-10T00:49:25.775 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:49:25 smithi170 ceph-mon[27620]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-10T00:49:25.775 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:49:25 smithi170 ceph-mon[27620]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:49:25.775 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:49:25 smithi170 ceph-mon[27620]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:49:25.775 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:49:25 smithi170 ceph-mon[27620]: Detected new or changed devices on smithi106 2024-03-10T00:49:25.775 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:49:25 smithi170 ceph-mon[27620]: pgmap v80: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-10T00:49:25.775 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:49:25 smithi170 ceph-mon[27620]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:49:25.775 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:49:25 smithi170 ceph-mon[27620]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-10T00:49:25.775 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:49:25 smithi170 ceph-mon[27620]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-10T00:49:25.775 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:49:25 smithi170 ceph-mon[27620]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-10T00:49:25.775 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:49:25 smithi170 ceph-mon[27620]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:49:25.776 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:49:25 smithi170 ceph-mon[27620]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-03-10T00:49:26.339 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:49:26 smithi106 ceph-mon[25742]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-03-10T00:49:26.339 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:49:26 smithi106 ceph-mon[25742]: osdmap e19: 3 total, 3 up, 3 in 2024-03-10T00:49:26.339 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:49:26 smithi106 ceph-mon[25742]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-03-10T00:49:26.339 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:49:26 smithi106 ceph-mon[33460]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-03-10T00:49:26.339 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:49:26 smithi106 ceph-mon[33460]: osdmap e19: 3 total, 3 up, 3 in 2024-03-10T00:49:26.339 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:49:26 smithi106 ceph-mon[33460]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-03-10T00:49:26.774 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:49:26 smithi170 ceph-mon[27620]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-03-10T00:49:26.775 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:49:26 smithi170 ceph-mon[27620]: osdmap e19: 3 total, 3 up, 3 in 2024-03-10T00:49:26.775 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:49:26 smithi170 ceph-mon[27620]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-03-10T00:49:26.801 INFO:teuthology.orchestra.run.smithi106.stderr:--> Zapping: /dev/nvme1n1 2024-03-10T00:49:26.801 INFO:teuthology.orchestra.run.smithi106.stderr:--> --destroy was not specified, but zapping a whole device will remove the partition table 2024-03-10T00:49:26.801 INFO:teuthology.orchestra.run.smithi106.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/nvme1n1 bs=1M count=10 conv=fsync 2024-03-10T00:49:26.801 INFO:teuthology.orchestra.run.smithi106.stderr: stderr: 10+0 records in 2024-03-10T00:49:26.801 INFO:teuthology.orchestra.run.smithi106.stderr:10+0 records out 2024-03-10T00:49:26.801 INFO:teuthology.orchestra.run.smithi106.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0131477 s, 798 MB/s 2024-03-10T00:49:26.802 INFO:teuthology.orchestra.run.smithi106.stderr:--> Zapping successful for: 2024-03-10T00:49:27.336 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:831bc188fa69f143bbba69ac76119e784500e8e3 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 75c37498-de77-11ee-95c7-87774f69a715 -- ceph orch daemon add osd smithi106:/dev/nvme1n1 2024-03-10T00:49:27.579 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/75c37498-de77-11ee-95c7-87774f69a715/mon.a/config 2024-03-10T00:49:27.588 INFO:journalctl@ceph.osd.0.smithi106.stdout:Mar 10 00:49:27 smithi106 sudo[48032]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme4n1 2024-03-10T00:49:27.588 INFO:journalctl@ceph.osd.0.smithi106.stdout:Mar 10 00:49:27 smithi106 sudo[48032]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-10T00:49:27.588 INFO:journalctl@ceph.osd.0.smithi106.stdout:Mar 10 00:49:27 smithi106 sudo[48032]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-10T00:49:27.589 INFO:journalctl@ceph.osd.0.smithi106.stdout:Mar 10 00:49:27 smithi106 sudo[48032]: pam_unix(sudo:session): session closed for user root 2024-03-10T00:49:27.589 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:49:27 smithi106 ceph-mon[33460]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-03-10T00:49:27.589 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:49:27 smithi106 ceph-mon[33460]: osdmap e20: 3 total, 3 up, 3 in 2024-03-10T00:49:27.589 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:49:27 smithi106 ceph-mon[33460]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:49:27.589 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:49:27 smithi106 ceph-mon[33460]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:49:27.589 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:49:27 smithi106 ceph-mon[33460]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:49:27.589 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:49:27 smithi106 ceph-mon[33460]: pgmap v83: 1 pgs: 1 unknown; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2024-03-10T00:49:27.589 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:49:27 smithi106 ceph-mon[25742]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-03-10T00:49:27.589 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:49:27 smithi106 ceph-mon[25742]: osdmap e20: 3 total, 3 up, 3 in 2024-03-10T00:49:27.590 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:49:27 smithi106 ceph-mon[25742]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:49:27.590 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:49:27 smithi106 ceph-mon[25742]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:49:27.590 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:49:27 smithi106 ceph-mon[25742]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:49:27.590 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:49:27 smithi106 ceph-mon[25742]: pgmap v83: 1 pgs: 1 unknown; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2024-03-10T00:49:27.775 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:49:27 smithi170 ceph-mon[27620]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-03-10T00:49:27.775 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:49:27 smithi170 ceph-mon[27620]: osdmap e20: 3 total, 3 up, 3 in 2024-03-10T00:49:27.775 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:49:27 smithi170 ceph-mon[27620]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:49:27.775 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:49:27 smithi170 ceph-mon[27620]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:49:27.775 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:49:27 smithi170 ceph-mon[27620]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:49:27.775 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:49:27 smithi170 ceph-mon[27620]: pgmap v83: 1 pgs: 1 unknown; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2024-03-10T00:49:27.960 INFO:journalctl@ceph.osd.0.smithi106.stdout:Mar 10 00:49:27 smithi106 sudo[48088]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme linux smart-log-add --json /dev/nvme4n1 2024-03-10T00:49:27.961 INFO:journalctl@ceph.osd.0.smithi106.stdout:Mar 10 00:49:27 smithi106 sudo[48088]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-10T00:49:27.961 INFO:journalctl@ceph.osd.0.smithi106.stdout:Mar 10 00:49:27 smithi106 sudo[48088]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-10T00:49:27.961 INFO:journalctl@ceph.osd.0.smithi106.stdout:Mar 10 00:49:27 smithi106 sudo[48088]: pam_unix(sudo:session): session closed for user root 2024-03-10T00:49:28.245 INFO:journalctl@ceph.osd.1.smithi106.stdout:Mar 10 00:49:27 smithi106 sudo[48128]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme3n1 2024-03-10T00:49:28.245 INFO:journalctl@ceph.osd.1.smithi106.stdout:Mar 10 00:49:27 smithi106 sudo[48128]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-10T00:49:28.245 INFO:journalctl@ceph.osd.1.smithi106.stdout:Mar 10 00:49:27 smithi106 sudo[48128]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-10T00:49:28.246 INFO:journalctl@ceph.osd.1.smithi106.stdout:Mar 10 00:49:28 smithi106 sudo[48128]: pam_unix(sudo:session): session closed for user root 2024-03-10T00:49:28.516 INFO:journalctl@ceph.osd.1.smithi106.stdout:Mar 10 00:49:28 smithi106 sudo[48132]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme linux smart-log-add --json /dev/nvme3n1 2024-03-10T00:49:28.516 INFO:journalctl@ceph.osd.1.smithi106.stdout:Mar 10 00:49:28 smithi106 sudo[48132]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-10T00:49:28.516 INFO:journalctl@ceph.osd.1.smithi106.stdout:Mar 10 00:49:28 smithi106 sudo[48132]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-10T00:49:28.516 INFO:journalctl@ceph.osd.1.smithi106.stdout:Mar 10 00:49:28 smithi106 sudo[48132]: pam_unix(sudo:session): session closed for user root 2024-03-10T00:49:28.716 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:49:28 smithi170 ceph-mon[27620]: osdmap e21: 3 total, 3 up, 3 in 2024-03-10T00:49:28.716 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:49:28 smithi170 ceph-mon[27620]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:49:28.716 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:49:28 smithi170 ceph-mon[27620]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:49:28.777 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:49:28 smithi106 ceph-mon[25742]: osdmap e21: 3 total, 3 up, 3 in 2024-03-10T00:49:28.778 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:49:28 smithi106 ceph-mon[25742]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:49:28.778 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:49:28 smithi106 ceph-mon[25742]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:49:28.778 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:49:28 smithi106 ceph-mon[33460]: osdmap e21: 3 total, 3 up, 3 in 2024-03-10T00:49:28.778 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:49:28 smithi106 ceph-mon[33460]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:49:28.778 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:49:28 smithi106 ceph-mon[33460]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:49:28.779 INFO:journalctl@ceph.osd.2.smithi106.stdout:Mar 10 00:49:28 smithi106 sudo[48144]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme2n1 2024-03-10T00:49:28.779 INFO:journalctl@ceph.osd.2.smithi106.stdout:Mar 10 00:49:28 smithi106 sudo[48144]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-10T00:49:28.779 INFO:journalctl@ceph.osd.2.smithi106.stdout:Mar 10 00:49:28 smithi106 sudo[48144]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-10T00:49:28.779 INFO:journalctl@ceph.osd.2.smithi106.stdout:Mar 10 00:49:28 smithi106 sudo[48144]: pam_unix(sudo:session): session closed for user root 2024-03-10T00:49:29.032 INFO:journalctl@ceph.osd.2.smithi106.stdout:Mar 10 00:49:28 smithi106 sudo[48151]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme linux smart-log-add --json /dev/nvme2n1 2024-03-10T00:49:29.032 INFO:journalctl@ceph.osd.2.smithi106.stdout:Mar 10 00:49:28 smithi106 sudo[48151]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-10T00:49:29.033 INFO:journalctl@ceph.osd.2.smithi106.stdout:Mar 10 00:49:28 smithi106 sudo[48151]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-10T00:49:29.033 INFO:journalctl@ceph.osd.2.smithi106.stdout:Mar 10 00:49:28 smithi106 sudo[48151]: pam_unix(sudo:session): session closed for user root 2024-03-10T00:49:29.324 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:49:29 smithi106 sudo[48184]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-10T00:49:29.324 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:49:29 smithi106 sudo[48184]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-10T00:49:29.324 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:49:29 smithi106 sudo[48184]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-10T00:49:29.324 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:49:29 smithi106 sudo[48184]: pam_unix(sudo:session): session closed for user root 2024-03-10T00:49:29.588 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:49:29 smithi106 sudo[48219]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-10T00:49:29.588 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:49:29 smithi106 sudo[48219]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-10T00:49:29.589 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:49:29 smithi106 sudo[48219]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-10T00:49:29.589 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:49:29 smithi106 sudo[48219]: pam_unix(sudo:session): session closed for user root 2024-03-10T00:49:29.864 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:49:29 smithi170 sudo[32081]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-10T00:49:29.865 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:49:29 smithi170 sudo[32081]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-10T00:49:29.865 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:49:29 smithi170 sudo[32081]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-10T00:49:29.865 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:49:29 smithi170 sudo[32081]: pam_unix(sudo:session): session closed for user root 2024-03-10T00:49:30.149 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:49:29 smithi106 ceph-mon[33460]: mgrmap e15: y(active, since 2m), standbys: x 2024-03-10T00:49:30.149 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:49:29 smithi106 ceph-mon[33460]: pgmap v85: 1 pgs: 1 unknown; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2024-03-10T00:49:30.149 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:49:29 smithi106 ceph-mon[33460]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:49:30.149 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:49:29 smithi106 ceph-mon[33460]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-10T00:49:30.149 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:49:29 smithi106 ceph-mon[33460]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-10T00:49:30.150 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:49:29 smithi106 ceph-mon[33460]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-10T00:49:30.150 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:49:29 smithi106 ceph-mon[33460]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-10T00:49:30.150 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:49:29 smithi106 ceph-mon[33460]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-10T00:49:30.150 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:49:29 smithi106 ceph-mon[33460]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:49:30.150 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:49:29 smithi106 ceph-mon[33460]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-10T00:49:30.150 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:49:29 smithi106 ceph-mon[33460]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-10T00:49:30.150 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:49:29 smithi106 ceph-mon[33460]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-10T00:49:30.150 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:49:29 smithi106 ceph-mon[33460]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-10T00:49:30.150 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:49:29 smithi106 ceph-mon[33460]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-10T00:49:30.150 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:49:29 smithi106 ceph-mon[33460]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-10T00:49:30.150 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:49:29 smithi106 ceph-mon[25742]: mgrmap e15: y(active, since 2m), standbys: x 2024-03-10T00:49:30.151 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:49:29 smithi106 ceph-mon[25742]: pgmap v85: 1 pgs: 1 unknown; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2024-03-10T00:49:30.151 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:49:29 smithi106 ceph-mon[25742]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:49:30.151 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:49:29 smithi106 ceph-mon[25742]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-10T00:49:30.151 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:49:29 smithi106 ceph-mon[25742]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-10T00:49:30.151 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:49:29 smithi106 ceph-mon[25742]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-10T00:49:30.151 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:49:29 smithi106 ceph-mon[25742]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-10T00:49:30.151 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:49:29 smithi106 ceph-mon[25742]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-10T00:49:30.151 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:49:29 smithi106 ceph-mon[25742]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:49:30.151 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:49:29 smithi106 ceph-mon[25742]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-10T00:49:30.151 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:49:29 smithi106 ceph-mon[25742]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-10T00:49:30.151 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:49:29 smithi106 ceph-mon[25742]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-10T00:49:30.151 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:49:29 smithi106 ceph-mon[25742]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-10T00:49:30.151 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:49:29 smithi106 ceph-mon[25742]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-10T00:49:30.151 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:49:29 smithi106 ceph-mon[25742]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-10T00:49:30.274 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:49:29 smithi170 sudo[32118]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-10T00:49:30.275 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:49:29 smithi170 sudo[32118]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-10T00:49:30.275 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:49:29 smithi170 sudo[32118]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-10T00:49:30.275 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:49:29 smithi170 sudo[32118]: pam_unix(sudo:session): session closed for user root 2024-03-10T00:49:30.275 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:49:29 smithi170 ceph-mon[27620]: mgrmap e15: y(active, since 2m), standbys: x 2024-03-10T00:49:30.275 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:49:29 smithi170 ceph-mon[27620]: pgmap v85: 1 pgs: 1 unknown; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2024-03-10T00:49:30.275 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:49:29 smithi170 ceph-mon[27620]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:49:30.275 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:49:29 smithi170 ceph-mon[27620]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-10T00:49:30.275 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:49:29 smithi170 ceph-mon[27620]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-10T00:49:30.275 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:49:29 smithi170 ceph-mon[27620]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-10T00:49:30.275 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:49:29 smithi170 ceph-mon[27620]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-10T00:49:30.276 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:49:29 smithi170 ceph-mon[27620]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-10T00:49:30.276 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:49:29 smithi170 ceph-mon[27620]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:49:30.276 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:49:29 smithi170 ceph-mon[27620]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-10T00:49:30.276 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:49:29 smithi170 ceph-mon[27620]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-10T00:49:30.276 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:49:29 smithi170 ceph-mon[27620]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-10T00:49:30.276 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:49:29 smithi170 ceph-mon[27620]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-10T00:49:30.276 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:49:29 smithi170 ceph-mon[27620]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-10T00:49:30.276 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:49:29 smithi170 ceph-mon[27620]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-10T00:49:30.412 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:49:30 smithi106 sudo[48344]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-10T00:49:30.412 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:49:30 smithi106 sudo[48344]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-10T00:49:30.413 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:49:30 smithi106 sudo[48344]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-10T00:49:30.413 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:49:30 smithi106 sudo[48344]: pam_unix(sudo:session): session closed for user root 2024-03-10T00:49:30.832 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:49:30 smithi106 sudo[48385]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-10T00:49:30.832 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:49:30 smithi106 sudo[48385]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-10T00:49:30.832 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:49:30 smithi106 sudo[48385]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-10T00:49:30.832 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:49:30 smithi106 sudo[48385]: pam_unix(sudo:session): session closed for user root 2024-03-10T00:49:31.088 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:49:30 smithi106 ceph-mon[25742]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-10T00:49:31.089 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:49:30 smithi106 ceph-mon[25742]: from='client.24209 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi106:/dev/nvme1n1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-10T00:49:31.089 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:49:30 smithi106 ceph-mon[25742]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-10T00:49:31.089 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:49:30 smithi106 ceph-mon[25742]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:49:31.089 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:49:30 smithi106 ceph-mon[25742]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-10T00:49:31.089 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:49:30 smithi106 ceph-mon[25742]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-10T00:49:31.089 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:49:30 smithi106 ceph-mon[25742]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-10T00:49:31.089 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:49:30 smithi106 ceph-mon[25742]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-10T00:49:31.089 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:49:30 smithi106 ceph-mon[25742]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-10T00:49:31.089 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:49:30 smithi106 ceph-mon[33460]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-10T00:49:31.089 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:49:30 smithi106 ceph-mon[33460]: from='client.24209 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi106:/dev/nvme1n1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-10T00:49:31.090 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:49:30 smithi106 ceph-mon[33460]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-10T00:49:31.090 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:49:30 smithi106 ceph-mon[33460]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:49:31.090 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:49:30 smithi106 ceph-mon[33460]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-10T00:49:31.090 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:49:30 smithi106 ceph-mon[33460]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-10T00:49:31.090 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:49:30 smithi106 ceph-mon[33460]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-10T00:49:31.090 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:49:30 smithi106 ceph-mon[33460]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-10T00:49:31.090 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:49:30 smithi106 ceph-mon[33460]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-10T00:49:31.275 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:49:30 smithi170 ceph-mon[27620]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-10T00:49:31.275 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:49:30 smithi170 ceph-mon[27620]: from='client.24209 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi106:/dev/nvme1n1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-10T00:49:31.275 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:49:30 smithi170 ceph-mon[27620]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-10T00:49:31.275 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:49:30 smithi170 ceph-mon[27620]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:49:31.275 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:49:30 smithi170 ceph-mon[27620]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-10T00:49:31.275 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:49:30 smithi170 ceph-mon[27620]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-10T00:49:31.275 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:49:30 smithi170 ceph-mon[27620]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-10T00:49:31.276 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:49:30 smithi170 ceph-mon[27620]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-10T00:49:31.276 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:49:30 smithi170 ceph-mon[27620]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-10T00:49:32.274 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:49:31 smithi170 ceph-mon[27620]: pgmap v86: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-03-10T00:49:32.339 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:49:31 smithi106 ceph-mon[25742]: pgmap v86: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-03-10T00:49:32.339 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:49:31 smithi106 ceph-mon[33460]: pgmap v86: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-03-10T00:49:33.274 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:49:32 smithi170 ceph-mon[27620]: from='client.? 172.21.15.106:0/3695613914' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b548c1c5-2658-47f6-89a3-f52f3e8c0758"}]: dispatch 2024-03-10T00:49:33.274 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:49:32 smithi170 ceph-mon[27620]: from='client.? 172.21.15.106:0/3695613914' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "b548c1c5-2658-47f6-89a3-f52f3e8c0758"}]': finished 2024-03-10T00:49:33.275 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:49:32 smithi170 ceph-mon[27620]: osdmap e22: 4 total, 3 up, 4 in 2024-03-10T00:49:33.275 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:49:32 smithi170 ceph-mon[27620]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-10T00:49:33.339 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:49:32 smithi106 ceph-mon[25742]: from='client.? 172.21.15.106:0/3695613914' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b548c1c5-2658-47f6-89a3-f52f3e8c0758"}]: dispatch 2024-03-10T00:49:33.339 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:49:32 smithi106 ceph-mon[25742]: from='client.? 172.21.15.106:0/3695613914' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "b548c1c5-2658-47f6-89a3-f52f3e8c0758"}]': finished 2024-03-10T00:49:33.339 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:49:32 smithi106 ceph-mon[25742]: osdmap e22: 4 total, 3 up, 4 in 2024-03-10T00:49:33.339 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:49:32 smithi106 ceph-mon[25742]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-10T00:49:33.339 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:49:32 smithi106 ceph-mon[33460]: from='client.? 172.21.15.106:0/3695613914' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b548c1c5-2658-47f6-89a3-f52f3e8c0758"}]: dispatch 2024-03-10T00:49:33.339 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:49:32 smithi106 ceph-mon[33460]: from='client.? 172.21.15.106:0/3695613914' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "b548c1c5-2658-47f6-89a3-f52f3e8c0758"}]': finished 2024-03-10T00:49:33.340 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:49:32 smithi106 ceph-mon[33460]: osdmap e22: 4 total, 3 up, 4 in 2024-03-10T00:49:33.340 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:49:32 smithi106 ceph-mon[33460]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-10T00:49:34.274 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:49:33 smithi170 ceph-mon[27620]: pgmap v88: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-03-10T00:49:34.275 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:49:33 smithi170 ceph-mon[27620]: from='client.? 172.21.15.106:0/4094509391' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-10T00:49:34.338 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:49:33 smithi106 ceph-mon[25742]: pgmap v88: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-03-10T00:49:34.338 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:49:33 smithi106 ceph-mon[25742]: from='client.? 172.21.15.106:0/4094509391' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-10T00:49:34.339 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:49:33 smithi106 ceph-mon[33460]: pgmap v88: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-03-10T00:49:34.339 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:49:33 smithi106 ceph-mon[33460]: from='client.? 172.21.15.106:0/4094509391' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-10T00:49:36.089 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:49:35 smithi106 ceph-mon[25742]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 91 MiB used, 268 GiB / 268 GiB avail; 6.7 KiB/s rd, 326 KiB/s wr, 21 op/s 2024-03-10T00:49:36.089 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:49:35 smithi106 ceph-mon[33460]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 91 MiB used, 268 GiB / 268 GiB avail; 6.7 KiB/s rd, 326 KiB/s wr, 21 op/s 2024-03-10T00:49:36.274 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:49:35 smithi170 ceph-mon[27620]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 91 MiB used, 268 GiB / 268 GiB avail; 6.7 KiB/s rd, 326 KiB/s wr, 21 op/s 2024-03-10T00:49:38.210 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:49:37 smithi106 ceph-mon[25742]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 91 MiB used, 268 GiB / 268 GiB avail; 4.5 KiB/s rd, 218 KiB/s wr, 14 op/s 2024-03-10T00:49:38.210 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:49:37 smithi106 ceph-mon[33460]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 91 MiB used, 268 GiB / 268 GiB avail; 4.5 KiB/s rd, 218 KiB/s wr, 14 op/s 2024-03-10T00:49:38.274 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:49:37 smithi170 ceph-mon[27620]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 91 MiB used, 268 GiB / 268 GiB avail; 4.5 KiB/s rd, 218 KiB/s wr, 14 op/s 2024-03-10T00:49:40.274 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:49:39 smithi170 ceph-mon[27620]: pgmap v91: 1 pgs: 1 active+clean; 577 KiB data, 91 MiB used, 268 GiB / 268 GiB avail; 3.4 KiB/s rd, 163 KiB/s wr, 10 op/s 2024-03-10T00:49:40.338 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:49:39 smithi106 ceph-mon[25742]: pgmap v91: 1 pgs: 1 active+clean; 577 KiB data, 91 MiB used, 268 GiB / 268 GiB avail; 3.4 KiB/s rd, 163 KiB/s wr, 10 op/s 2024-03-10T00:49:40.339 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:49:39 smithi106 ceph-mon[33460]: pgmap v91: 1 pgs: 1 active+clean; 577 KiB data, 91 MiB used, 268 GiB / 268 GiB avail; 3.4 KiB/s rd, 163 KiB/s wr, 10 op/s 2024-03-10T00:49:42.067 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:49:41 smithi106 ceph-mon[25742]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 91 MiB used, 268 GiB / 268 GiB avail; 2.7 KiB/s rd, 131 KiB/s wr, 8 op/s 2024-03-10T00:49:42.067 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:49:41 smithi106 ceph-mon[25742]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:49:42.068 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:49:41 smithi106 ceph-mon[25742]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:49:42.068 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:49:41 smithi106 ceph-mon[25742]: Detected new or changed devices on smithi106 2024-03-10T00:49:42.068 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:49:41 smithi106 ceph-mon[25742]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:49:42.068 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:49:41 smithi106 ceph-mon[33460]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 91 MiB used, 268 GiB / 268 GiB avail; 2.7 KiB/s rd, 131 KiB/s wr, 8 op/s 2024-03-10T00:49:42.068 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:49:41 smithi106 ceph-mon[33460]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:49:42.068 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:49:41 smithi106 ceph-mon[33460]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:49:42.068 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:49:41 smithi106 ceph-mon[33460]: Detected new or changed devices on smithi106 2024-03-10T00:49:42.068 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:49:41 smithi106 ceph-mon[33460]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:49:42.219 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:49:41 smithi170 ceph-mon[27620]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 91 MiB used, 268 GiB / 268 GiB avail; 2.7 KiB/s rd, 131 KiB/s wr, 8 op/s 2024-03-10T00:49:42.219 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:49:41 smithi170 ceph-mon[27620]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:49:42.219 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:49:41 smithi170 ceph-mon[27620]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:49:42.219 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:49:41 smithi170 ceph-mon[27620]: Detected new or changed devices on smithi106 2024-03-10T00:49:42.219 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:49:41 smithi170 ceph-mon[27620]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:49:43.028 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:49:42 smithi106 ceph-mon[25742]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-03-10T00:49:43.028 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:49:42 smithi106 ceph-mon[25742]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-10T00:49:43.028 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:49:42 smithi106 ceph-mon[33460]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-03-10T00:49:43.028 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:49:42 smithi106 ceph-mon[33460]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-10T00:49:43.274 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:49:42 smithi170 ceph-mon[27620]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-03-10T00:49:43.275 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:49:42 smithi170 ceph-mon[27620]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-10T00:49:44.067 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:49:43 smithi106 ceph-mon[33460]: Deploying daemon osd.3 on smithi106 2024-03-10T00:49:44.067 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:49:43 smithi106 ceph-mon[33460]: pgmap v93: 1 pgs: 1 active+clean; 577 KiB data, 91 MiB used, 268 GiB / 268 GiB avail; 2.6 KiB/s rd, 124 KiB/s wr, 8 op/s 2024-03-10T00:49:44.068 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:49:43 smithi106 ceph-mon[25742]: Deploying daemon osd.3 on smithi106 2024-03-10T00:49:44.068 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:49:43 smithi106 ceph-mon[25742]: pgmap v93: 1 pgs: 1 active+clean; 577 KiB data, 91 MiB used, 268 GiB / 268 GiB avail; 2.6 KiB/s rd, 124 KiB/s wr, 8 op/s 2024-03-10T00:49:44.274 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:49:43 smithi170 ceph-mon[27620]: Deploying daemon osd.3 on smithi106 2024-03-10T00:49:44.274 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:49:43 smithi170 ceph-mon[27620]: pgmap v93: 1 pgs: 1 active+clean; 577 KiB data, 91 MiB used, 268 GiB / 268 GiB avail; 2.6 KiB/s rd, 124 KiB/s wr, 8 op/s 2024-03-10T00:49:46.274 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:49:45 smithi170 ceph-mon[27620]: pgmap v94: 1 pgs: 1 active+clean; 577 KiB data, 91 MiB used, 268 GiB / 268 GiB avail; 2.2 KiB/s rd, 109 KiB/s wr, 7 op/s 2024-03-10T00:49:46.339 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:49:45 smithi106 ceph-mon[33460]: pgmap v94: 1 pgs: 1 active+clean; 577 KiB data, 91 MiB used, 268 GiB / 268 GiB avail; 2.2 KiB/s rd, 109 KiB/s wr, 7 op/s 2024-03-10T00:49:46.339 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:49:45 smithi106 ceph-mon[25742]: pgmap v94: 1 pgs: 1 active+clean; 577 KiB data, 91 MiB used, 268 GiB / 268 GiB avail; 2.2 KiB/s rd, 109 KiB/s wr, 7 op/s 2024-03-10T00:49:47.875 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:49:47 smithi106 ceph-mon[25742]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:49:47.875 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:49:47 smithi106 ceph-mon[25742]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:49:47.875 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:49:47 smithi106 ceph-mon[25742]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:49:47.875 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:49:47 smithi106 ceph-mon[25742]: pgmap v95: 1 pgs: 1 active+clean; 577 KiB data, 91 MiB used, 268 GiB / 268 GiB avail 2024-03-10T00:49:47.875 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:49:47 smithi106 ceph-mon[33460]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:49:47.876 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:49:47 smithi106 ceph-mon[33460]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:49:47.876 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:49:47 smithi106 ceph-mon[33460]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:49:47.876 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:49:47 smithi106 ceph-mon[33460]: pgmap v95: 1 pgs: 1 active+clean; 577 KiB data, 91 MiB used, 268 GiB / 268 GiB avail 2024-03-10T00:49:48.024 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:49:47 smithi170 ceph-mon[27620]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:49:48.025 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:49:47 smithi170 ceph-mon[27620]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:49:48.025 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:49:47 smithi170 ceph-mon[27620]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:49:48.025 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:49:47 smithi170 ceph-mon[27620]: pgmap v95: 1 pgs: 1 active+clean; 577 KiB data, 91 MiB used, 268 GiB / 268 GiB avail 2024-03-10T00:49:48.974 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:49:48 smithi170 ceph-mon[27620]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:49:48.974 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:49:48 smithi170 ceph-mon[27620]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:49:48.974 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:49:48 smithi170 ceph-mon[27620]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-10T00:49:48.975 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:49:48 smithi170 ceph-mon[27620]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-10T00:49:48.975 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:49:48 smithi170 ceph-mon[27620]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-10T00:49:48.975 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:49:48 smithi170 ceph-mon[27620]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:49:49.247 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:49:48 smithi106 ceph-mon[33460]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:49:49.247 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:49:48 smithi106 ceph-mon[33460]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:49:49.247 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:49:48 smithi106 ceph-mon[33460]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-10T00:49:49.247 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:49:48 smithi106 ceph-mon[33460]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-10T00:49:49.247 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:49:48 smithi106 ceph-mon[33460]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-10T00:49:49.247 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:49:48 smithi106 ceph-mon[33460]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:49:49.247 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:49:48 smithi106 ceph-mon[25742]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:49:49.247 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:49:48 smithi106 ceph-mon[25742]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:49:49.247 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:49:48 smithi106 ceph-mon[25742]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-10T00:49:49.248 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:49:48 smithi106 ceph-mon[25742]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-10T00:49:49.248 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:49:48 smithi106 ceph-mon[25742]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-10T00:49:49.248 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:49:48 smithi106 ceph-mon[25742]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:49:50.089 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:49:49 smithi106 ceph-mon[25742]: pgmap v96: 1 pgs: 1 active+clean; 577 KiB data, 91 MiB used, 268 GiB / 268 GiB avail 2024-03-10T00:49:50.089 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:49:49 smithi106 ceph-mon[33460]: pgmap v96: 1 pgs: 1 active+clean; 577 KiB data, 91 MiB used, 268 GiB / 268 GiB avail 2024-03-10T00:49:50.138 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:49:49 smithi170 ceph-mon[27620]: pgmap v96: 1 pgs: 1 active+clean; 577 KiB data, 91 MiB used, 268 GiB / 268 GiB avail 2024-03-10T00:49:51.193 INFO:teuthology.orchestra.run.smithi106.stdout:Created osd(s) 3 on host 'smithi106' 2024-03-10T00:49:51.851 DEBUG:teuthology.orchestra.run.smithi106:osd.3> sudo journalctl -f -n 0 -u ceph-75c37498-de77-11ee-95c7-87774f69a715@osd.3.service 2024-03-10T00:49:51.854 INFO:tasks.cephadm:Deploying osd.4 on smithi170 with /dev/nvme4n1... 2024-03-10T00:49:51.854 DEBUG:teuthology.orchestra.run.smithi170:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:831bc188fa69f143bbba69ac76119e784500e8e3 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 75c37498-de77-11ee-95c7-87774f69a715 -- ceph-volume lvm zap /dev/nvme4n1 2024-03-10T00:49:52.025 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:49:51 smithi170 ceph-mon[27620]: pgmap v97: 1 pgs: 1 active+clean; 577 KiB data, 91 MiB used, 268 GiB / 268 GiB avail 2024-03-10T00:49:52.025 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:49:51 smithi170 ceph-mon[27620]: from='osd.3 [v2:172.21.15.106:6826/2289173680,v1:172.21.15.106:6827/2289173680]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-03-10T00:49:52.025 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:49:51 smithi170 ceph-mon[27620]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:49:52.025 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:49:51 smithi170 ceph-mon[27620]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:49:52.025 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:49:51 smithi170 ceph-mon[27620]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-10T00:49:52.025 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:49:51 smithi170 ceph-mon[27620]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-10T00:49:52.025 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:49:51 smithi170 ceph-mon[27620]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-10T00:49:52.025 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:49:51 smithi170 ceph-mon[27620]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-10T00:49:52.025 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:49:51 smithi170 ceph-mon[27620]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-10T00:49:52.025 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:49:51 smithi170 ceph-mon[27620]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:49:52.025 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:49:51 smithi170 ceph-mon[27620]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-10T00:49:52.025 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:49:51 smithi170 ceph-mon[27620]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-10T00:49:52.025 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:49:51 smithi170 ceph-mon[27620]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:49:52.089 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:49:51 smithi106 ceph-mon[25742]: pgmap v97: 1 pgs: 1 active+clean; 577 KiB data, 91 MiB used, 268 GiB / 268 GiB avail 2024-03-10T00:49:52.089 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:49:51 smithi106 ceph-mon[25742]: from='osd.3 [v2:172.21.15.106:6826/2289173680,v1:172.21.15.106:6827/2289173680]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-03-10T00:49:52.089 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:49:51 smithi106 ceph-mon[25742]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:49:52.089 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:49:51 smithi106 ceph-mon[25742]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:49:52.089 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:49:51 smithi106 ceph-mon[25742]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-10T00:49:52.089 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:49:51 smithi106 ceph-mon[25742]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-10T00:49:52.089 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:49:51 smithi106 ceph-mon[25742]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-10T00:49:52.089 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:49:51 smithi106 ceph-mon[25742]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-10T00:49:52.089 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:49:51 smithi106 ceph-mon[25742]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-10T00:49:52.089 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:49:51 smithi106 ceph-mon[25742]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:49:52.089 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:49:51 smithi106 ceph-mon[25742]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-10T00:49:52.090 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:49:51 smithi106 ceph-mon[25742]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-10T00:49:52.090 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:49:51 smithi106 ceph-mon[25742]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:49:52.090 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:49:51 smithi106 ceph-mon[33460]: pgmap v97: 1 pgs: 1 active+clean; 577 KiB data, 91 MiB used, 268 GiB / 268 GiB avail 2024-03-10T00:49:52.090 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:49:51 smithi106 ceph-mon[33460]: from='osd.3 [v2:172.21.15.106:6826/2289173680,v1:172.21.15.106:6827/2289173680]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-03-10T00:49:52.090 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:49:51 smithi106 ceph-mon[33460]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:49:52.090 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:49:51 smithi106 ceph-mon[33460]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:49:52.090 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:49:51 smithi106 ceph-mon[33460]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-10T00:49:52.090 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:49:51 smithi106 ceph-mon[33460]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-10T00:49:52.090 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:49:51 smithi106 ceph-mon[33460]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-10T00:49:52.090 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:49:51 smithi106 ceph-mon[33460]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-10T00:49:52.090 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:49:51 smithi106 ceph-mon[33460]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-10T00:49:52.091 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:49:51 smithi106 ceph-mon[33460]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:49:52.091 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:49:51 smithi106 ceph-mon[33460]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-10T00:49:52.091 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:49:51 smithi106 ceph-mon[33460]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-10T00:49:52.091 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:49:51 smithi106 ceph-mon[33460]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:49:52.102 INFO:teuthology.orchestra.run.smithi170.stderr:Inferring config /var/lib/ceph/75c37498-de77-11ee-95c7-87774f69a715/mon.b/config 2024-03-10T00:49:53.256 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:49:53 smithi170 ceph-mon[27620]: Adjusting osd_memory_target on smithi106 to 3760M 2024-03-10T00:49:53.256 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:49:53 smithi170 ceph-mon[27620]: from='osd.3 [v2:172.21.15.106:6826/2289173680,v1:172.21.15.106:6827/2289173680]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-03-10T00:49:53.256 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:49:53 smithi170 ceph-mon[27620]: osdmap e23: 4 total, 3 up, 4 in 2024-03-10T00:49:53.256 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:49:53 smithi170 ceph-mon[27620]: from='osd.3 [v2:172.21.15.106:6826/2289173680,v1:172.21.15.106:6827/2289173680]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi106", "root=default"]}]: dispatch 2024-03-10T00:49:53.256 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:49:53 smithi170 ceph-mon[27620]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-10T00:49:53.256 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:49:53 smithi170 ceph-mon[27620]: from='osd.3 [v2:172.21.15.106:6826/2289173680,v1:172.21.15.106:6827/2289173680]' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi106", "root=default"]}]': finished 2024-03-10T00:49:53.387 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:49:53 smithi106 ceph-mon[33460]: Adjusting osd_memory_target on smithi106 to 3760M 2024-03-10T00:49:53.387 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:49:53 smithi106 ceph-mon[33460]: from='osd.3 [v2:172.21.15.106:6826/2289173680,v1:172.21.15.106:6827/2289173680]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-03-10T00:49:53.387 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:49:53 smithi106 ceph-mon[33460]: osdmap e23: 4 total, 3 up, 4 in 2024-03-10T00:49:53.387 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:49:53 smithi106 ceph-mon[33460]: from='osd.3 [v2:172.21.15.106:6826/2289173680,v1:172.21.15.106:6827/2289173680]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi106", "root=default"]}]: dispatch 2024-03-10T00:49:53.387 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:49:53 smithi106 ceph-mon[33460]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-10T00:49:53.387 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:49:53 smithi106 ceph-mon[33460]: from='osd.3 [v2:172.21.15.106:6826/2289173680,v1:172.21.15.106:6827/2289173680]' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi106", "root=default"]}]': finished 2024-03-10T00:49:53.388 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:49:53 smithi106 ceph-mon[25742]: Adjusting osd_memory_target on smithi106 to 3760M 2024-03-10T00:49:53.388 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:49:53 smithi106 ceph-mon[25742]: from='osd.3 [v2:172.21.15.106:6826/2289173680,v1:172.21.15.106:6827/2289173680]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-03-10T00:49:53.388 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:49:53 smithi106 ceph-mon[25742]: osdmap e23: 4 total, 3 up, 4 in 2024-03-10T00:49:53.388 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:49:53 smithi106 ceph-mon[25742]: from='osd.3 [v2:172.21.15.106:6826/2289173680,v1:172.21.15.106:6827/2289173680]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi106", "root=default"]}]: dispatch 2024-03-10T00:49:53.388 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:49:53 smithi106 ceph-mon[25742]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-10T00:49:53.388 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:49:53 smithi106 ceph-mon[25742]: from='osd.3 [v2:172.21.15.106:6826/2289173680,v1:172.21.15.106:6827/2289173680]' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi106", "root=default"]}]': finished 2024-03-10T00:49:53.838 INFO:journalctl@ceph.osd.3.smithi106.stdout:Mar 10 00:49:53 smithi106 ceph-75c37498-de77-11ee-95c7-87774f69a715-osd-3[50617]: 2024-03-10T00:49:53.388+0000 7f80a0515700 -1 osd.3 0 waiting for initial osdmap 2024-03-10T00:49:53.839 INFO:journalctl@ceph.osd.3.smithi106.stdout:Mar 10 00:49:53 smithi106 ceph-75c37498-de77-11ee-95c7-87774f69a715-osd-3[50617]: 2024-03-10T00:49:53.392+0000 7f8098300700 -1 osd.3 24 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-10T00:49:54.192 INFO:teuthology.orchestra.run.smithi170.stderr:--> Zapping: /dev/nvme4n1 2024-03-10T00:49:54.192 INFO:teuthology.orchestra.run.smithi170.stderr:--> --destroy was not specified, but zapping a whole device will remove the partition table 2024-03-10T00:49:54.192 INFO:teuthology.orchestra.run.smithi170.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/nvme4n1 bs=1M count=10 conv=fsync 2024-03-10T00:49:54.192 INFO:teuthology.orchestra.run.smithi170.stderr: stderr: 10+0 records in 2024-03-10T00:49:54.192 INFO:teuthology.orchestra.run.smithi170.stderr:10+0 records out 2024-03-10T00:49:54.192 INFO:teuthology.orchestra.run.smithi170.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.00942907 s, 1.1 GB/s 2024-03-10T00:49:54.192 INFO:teuthology.orchestra.run.smithi170.stderr:--> Zapping successful for: 2024-03-10T00:49:54.192 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:49:54 smithi170 ceph-mon[27620]: purged_snaps scrub starts 2024-03-10T00:49:54.192 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:49:54 smithi170 ceph-mon[27620]: purged_snaps scrub ok 2024-03-10T00:49:54.193 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:49:54 smithi170 ceph-mon[27620]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 91 MiB used, 268 GiB / 268 GiB avail 2024-03-10T00:49:54.193 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:49:54 smithi170 ceph-mon[27620]: osdmap e24: 4 total, 3 up, 4 in 2024-03-10T00:49:54.193 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:49:54 smithi170 ceph-mon[27620]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-10T00:49:54.193 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:49:54 smithi170 ceph-mon[27620]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-10T00:49:54.193 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:49:54 smithi170 ceph-mon[27620]: from='osd.3 [v2:172.21.15.106:6826/2289173680,v1:172.21.15.106:6827/2289173680]' entity='osd.3' 2024-03-10T00:49:54.193 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:49:54 smithi170 ceph-mon[27620]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-10T00:49:54.589 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:49:54 smithi106 ceph-mon[25742]: purged_snaps scrub starts 2024-03-10T00:49:54.589 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:49:54 smithi106 ceph-mon[25742]: purged_snaps scrub ok 2024-03-10T00:49:54.589 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:49:54 smithi106 ceph-mon[25742]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 91 MiB used, 268 GiB / 268 GiB avail 2024-03-10T00:49:54.589 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:49:54 smithi106 ceph-mon[25742]: osdmap e24: 4 total, 3 up, 4 in 2024-03-10T00:49:54.589 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:49:54 smithi106 ceph-mon[25742]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-10T00:49:54.589 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:49:54 smithi106 ceph-mon[25742]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-10T00:49:54.589 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:49:54 smithi106 ceph-mon[25742]: from='osd.3 [v2:172.21.15.106:6826/2289173680,v1:172.21.15.106:6827/2289173680]' entity='osd.3' 2024-03-10T00:49:54.589 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:49:54 smithi106 ceph-mon[25742]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-10T00:49:54.590 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:49:54 smithi106 ceph-mon[33460]: purged_snaps scrub starts 2024-03-10T00:49:54.590 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:49:54 smithi106 ceph-mon[33460]: purged_snaps scrub ok 2024-03-10T00:49:54.590 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:49:54 smithi106 ceph-mon[33460]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 91 MiB used, 268 GiB / 268 GiB avail 2024-03-10T00:49:54.590 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:49:54 smithi106 ceph-mon[33460]: osdmap e24: 4 total, 3 up, 4 in 2024-03-10T00:49:54.590 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:49:54 smithi106 ceph-mon[33460]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-10T00:49:54.590 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:49:54 smithi106 ceph-mon[33460]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-10T00:49:54.590 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:49:54 smithi106 ceph-mon[33460]: from='osd.3 [v2:172.21.15.106:6826/2289173680,v1:172.21.15.106:6827/2289173680]' entity='osd.3' 2024-03-10T00:49:54.590 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:49:54 smithi106 ceph-mon[33460]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-10T00:49:54.797 DEBUG:teuthology.orchestra.run.smithi170:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:831bc188fa69f143bbba69ac76119e784500e8e3 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 75c37498-de77-11ee-95c7-87774f69a715 -- ceph orch daemon add osd smithi170:/dev/nvme4n1 2024-03-10T00:49:55.032 INFO:teuthology.orchestra.run.smithi170.stderr:Inferring config /var/lib/ceph/75c37498-de77-11ee-95c7-87774f69a715/mon.b/config 2024-03-10T00:49:55.448 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:49:55 smithi170 ceph-mon[27620]: osd.3 [v2:172.21.15.106:6826/2289173680,v1:172.21.15.106:6827/2289173680] boot 2024-03-10T00:49:55.448 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:49:55 smithi170 ceph-mon[27620]: osdmap e25: 4 total, 4 up, 4 in 2024-03-10T00:49:55.448 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:49:55 smithi170 ceph-mon[27620]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-10T00:49:55.448 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:49:55 smithi170 ceph-mon[27620]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-03-10T00:49:55.838 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:49:55 smithi106 ceph-mon[25742]: osd.3 [v2:172.21.15.106:6826/2289173680,v1:172.21.15.106:6827/2289173680] boot 2024-03-10T00:49:55.839 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:49:55 smithi106 ceph-mon[25742]: osdmap e25: 4 total, 4 up, 4 in 2024-03-10T00:49:55.839 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:49:55 smithi106 ceph-mon[25742]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-10T00:49:55.839 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:49:55 smithi106 ceph-mon[25742]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-03-10T00:49:55.839 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:49:55 smithi106 ceph-mon[33460]: osd.3 [v2:172.21.15.106:6826/2289173680,v1:172.21.15.106:6827/2289173680] boot 2024-03-10T00:49:55.839 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:49:55 smithi106 ceph-mon[33460]: osdmap e25: 4 total, 4 up, 4 in 2024-03-10T00:49:55.839 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:49:55 smithi106 ceph-mon[33460]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-10T00:49:55.839 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:49:55 smithi106 ceph-mon[33460]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-03-10T00:49:56.445 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:49:56 smithi170 ceph-mon[27620]: osdmap e26: 4 total, 4 up, 4 in 2024-03-10T00:49:56.838 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:49:56 smithi106 ceph-mon[25742]: osdmap e26: 4 total, 4 up, 4 in 2024-03-10T00:49:56.839 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:49:56 smithi106 ceph-mon[33460]: osdmap e26: 4 total, 4 up, 4 in 2024-03-10T00:49:57.715 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:49:57 smithi170 ceph-mon[27620]: pgmap v104: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-03-10T00:49:57.715 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:49:57 smithi170 ceph-mon[27620]: from='client.24187 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi170:/dev/nvme4n1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-10T00:49:57.715 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:49:57 smithi170 ceph-mon[27620]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-10T00:49:57.715 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:49:57 smithi170 ceph-mon[27620]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-10T00:49:57.715 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:49:57 smithi170 ceph-mon[27620]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-10T00:49:57.753 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:49:57 smithi106 ceph-mon[25742]: pgmap v104: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-03-10T00:49:57.753 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:49:57 smithi106 ceph-mon[25742]: from='client.24187 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi170:/dev/nvme4n1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-10T00:49:57.753 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:49:57 smithi106 ceph-mon[25742]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-10T00:49:57.753 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:49:57 smithi106 ceph-mon[25742]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-10T00:49:57.754 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:49:57 smithi106 ceph-mon[25742]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-10T00:49:57.754 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:49:57 smithi106 ceph-mon[33460]: pgmap v104: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-03-10T00:49:57.754 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:49:57 smithi106 ceph-mon[33460]: from='client.24187 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi170:/dev/nvme4n1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-10T00:49:57.754 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:49:57 smithi106 ceph-mon[33460]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-10T00:49:57.754 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:49:57 smithi106 ceph-mon[33460]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-10T00:49:57.754 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:49:57 smithi106 ceph-mon[33460]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-10T00:50:00.089 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:49:59 smithi106 ceph-mon[25742]: pgmap v105: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-03-10T00:50:00.089 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:49:59 smithi106 ceph-mon[25742]: from='client.? 172.21.15.170:0/169661035' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "4fcacd32-e9ca-4aa7-b99e-bac33fc58754"}]: dispatch 2024-03-10T00:50:00.089 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:49:59 smithi106 ceph-mon[25742]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "4fcacd32-e9ca-4aa7-b99e-bac33fc58754"}]: dispatch 2024-03-10T00:50:00.089 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:49:59 smithi106 ceph-mon[25742]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "4fcacd32-e9ca-4aa7-b99e-bac33fc58754"}]': finished 2024-03-10T00:50:00.089 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:49:59 smithi106 ceph-mon[25742]: osdmap e27: 5 total, 4 up, 5 in 2024-03-10T00:50:00.089 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:49:59 smithi106 ceph-mon[25742]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-10T00:50:00.089 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:49:59 smithi106 ceph-mon[33460]: pgmap v105: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-03-10T00:50:00.090 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:49:59 smithi106 ceph-mon[33460]: from='client.? 172.21.15.170:0/169661035' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "4fcacd32-e9ca-4aa7-b99e-bac33fc58754"}]: dispatch 2024-03-10T00:50:00.090 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:49:59 smithi106 ceph-mon[33460]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "4fcacd32-e9ca-4aa7-b99e-bac33fc58754"}]: dispatch 2024-03-10T00:50:00.090 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:49:59 smithi106 ceph-mon[33460]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "4fcacd32-e9ca-4aa7-b99e-bac33fc58754"}]': finished 2024-03-10T00:50:00.090 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:49:59 smithi106 ceph-mon[33460]: osdmap e27: 5 total, 4 up, 5 in 2024-03-10T00:50:00.090 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:49:59 smithi106 ceph-mon[33460]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-10T00:50:00.119 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:49:59 smithi170 ceph-mon[27620]: pgmap v105: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-03-10T00:50:00.119 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:49:59 smithi170 ceph-mon[27620]: from='client.? 172.21.15.170:0/169661035' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "4fcacd32-e9ca-4aa7-b99e-bac33fc58754"}]: dispatch 2024-03-10T00:50:00.120 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:49:59 smithi170 ceph-mon[27620]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "4fcacd32-e9ca-4aa7-b99e-bac33fc58754"}]: dispatch 2024-03-10T00:50:00.120 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:49:59 smithi170 ceph-mon[27620]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "4fcacd32-e9ca-4aa7-b99e-bac33fc58754"}]': finished 2024-03-10T00:50:00.120 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:49:59 smithi170 ceph-mon[27620]: osdmap e27: 5 total, 4 up, 5 in 2024-03-10T00:50:00.120 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:49:59 smithi170 ceph-mon[27620]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-10T00:50:01.088 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:50:00 smithi106 ceph-mon[25742]: overall HEALTH_OK 2024-03-10T00:50:01.089 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:50:00 smithi106 ceph-mon[25742]: from='client.? 172.21.15.170:0/720809658' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-10T00:50:01.089 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:50:00 smithi106 ceph-mon[33460]: overall HEALTH_OK 2024-03-10T00:50:01.089 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:50:00 smithi106 ceph-mon[33460]: from='client.? 172.21.15.170:0/720809658' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-10T00:50:01.274 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:50:00 smithi170 ceph-mon[27620]: overall HEALTH_OK 2024-03-10T00:50:01.275 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:50:00 smithi170 ceph-mon[27620]: from='client.? 172.21.15.170:0/720809658' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-10T00:50:02.274 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:50:01 smithi170 ceph-mon[27620]: pgmap v107: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-03-10T00:50:02.274 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:50:01 smithi170 ceph-mon[27620]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:50:02.275 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:50:01 smithi170 ceph-mon[27620]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:50:02.275 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:50:01 smithi170 ceph-mon[27620]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:50:02.338 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:50:01 smithi106 ceph-mon[25742]: pgmap v107: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-03-10T00:50:02.339 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:50:01 smithi106 ceph-mon[25742]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:50:02.339 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:50:01 smithi106 ceph-mon[25742]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:50:02.339 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:50:01 smithi106 ceph-mon[25742]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:50:02.339 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:50:01 smithi106 ceph-mon[33460]: pgmap v107: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-03-10T00:50:02.339 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:50:01 smithi106 ceph-mon[33460]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:50:02.339 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:50:01 smithi106 ceph-mon[33460]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:50:02.339 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:50:01 smithi106 ceph-mon[33460]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:50:04.179 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:50:03 smithi170 ceph-mon[27620]: pgmap v108: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-03-10T00:50:04.338 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:50:03 smithi106 ceph-mon[25742]: pgmap v108: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-03-10T00:50:04.339 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:50:03 smithi106 ceph-mon[33460]: pgmap v108: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-03-10T00:50:06.141 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:50:05 smithi170 ceph-mon[27620]: pgmap v109: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-03-10T00:50:06.338 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:50:05 smithi106 ceph-mon[25742]: pgmap v109: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-03-10T00:50:06.339 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:50:05 smithi106 ceph-mon[33460]: pgmap v109: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-03-10T00:50:07.025 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:50:06 smithi170 ceph-mon[27620]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-03-10T00:50:07.025 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:50:06 smithi170 ceph-mon[27620]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-10T00:50:07.025 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:50:06 smithi170 ceph-mon[27620]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:50:07.025 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:50:06 smithi170 ceph-mon[27620]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:50:07.026 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:50:06 smithi170 ceph-mon[27620]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:50:07.338 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:50:06 smithi106 ceph-mon[25742]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-03-10T00:50:07.339 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:50:06 smithi106 ceph-mon[25742]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-10T00:50:07.339 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:50:06 smithi106 ceph-mon[25742]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:50:07.339 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:50:06 smithi106 ceph-mon[25742]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:50:07.339 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:50:06 smithi106 ceph-mon[25742]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:50:07.339 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:50:06 smithi106 ceph-mon[33460]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-03-10T00:50:07.339 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:50:06 smithi106 ceph-mon[33460]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-10T00:50:07.339 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:50:06 smithi106 ceph-mon[33460]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:50:07.339 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:50:06 smithi106 ceph-mon[33460]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:50:07.340 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:50:06 smithi106 ceph-mon[33460]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:50:07.964 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:50:07 smithi170 ceph-mon[27620]: Deploying daemon osd.4 on smithi170 2024-03-10T00:50:07.964 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:50:07 smithi170 ceph-mon[27620]: pgmap v110: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-03-10T00:50:08.339 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:50:07 smithi106 ceph-mon[25742]: Deploying daemon osd.4 on smithi170 2024-03-10T00:50:08.339 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:50:07 smithi106 ceph-mon[25742]: pgmap v110: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-03-10T00:50:08.340 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:50:07 smithi106 ceph-mon[33460]: Deploying daemon osd.4 on smithi170 2024-03-10T00:50:08.340 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:50:07 smithi106 ceph-mon[33460]: pgmap v110: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-03-10T00:50:10.025 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:50:09 smithi170 ceph-mon[27620]: pgmap v111: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-03-10T00:50:10.338 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:50:09 smithi106 ceph-mon[25742]: pgmap v111: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-03-10T00:50:10.339 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:50:09 smithi106 ceph-mon[33460]: pgmap v111: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-03-10T00:50:12.024 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:50:11 smithi170 ceph-mon[27620]: pgmap v112: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-03-10T00:50:12.339 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:50:11 smithi106 ceph-mon[25742]: pgmap v112: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-03-10T00:50:12.339 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:50:11 smithi106 ceph-mon[33460]: pgmap v112: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-03-10T00:50:14.024 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:50:13 smithi170 ceph-mon[27620]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:50:14.025 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:50:13 smithi170 ceph-mon[27620]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:50:14.025 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:50:13 smithi170 ceph-mon[27620]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-10T00:50:14.025 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:50:13 smithi170 ceph-mon[27620]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-10T00:50:14.025 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:50:13 smithi170 ceph-mon[27620]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-10T00:50:14.025 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:50:13 smithi170 ceph-mon[27620]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:50:14.025 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:50:13 smithi170 ceph-mon[27620]: pgmap v113: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-03-10T00:50:14.088 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:50:13 smithi106 ceph-mon[33460]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:50:14.088 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:50:13 smithi106 ceph-mon[33460]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:50:14.089 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:50:13 smithi106 ceph-mon[33460]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-10T00:50:14.089 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:50:13 smithi106 ceph-mon[33460]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-10T00:50:14.089 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:50:13 smithi106 ceph-mon[33460]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-10T00:50:14.089 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:50:13 smithi106 ceph-mon[33460]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:50:14.089 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:50:13 smithi106 ceph-mon[33460]: pgmap v113: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-03-10T00:50:14.089 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:50:13 smithi106 ceph-mon[25742]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:50:14.089 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:50:13 smithi106 ceph-mon[25742]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:50:14.089 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:50:13 smithi106 ceph-mon[25742]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-10T00:50:14.089 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:50:13 smithi106 ceph-mon[25742]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-10T00:50:14.089 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:50:13 smithi106 ceph-mon[25742]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-10T00:50:14.089 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:50:13 smithi106 ceph-mon[25742]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:50:14.090 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:50:13 smithi106 ceph-mon[25742]: pgmap v113: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-03-10T00:50:16.006 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:50:15 smithi170 ceph-mon[27620]: pgmap v114: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-03-10T00:50:16.006 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:50:15 smithi170 ceph-mon[27620]: from='osd.4 [v2:172.21.15.170:6800/335809953,v1:172.21.15.170:6801/335809953]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-10T00:50:16.006 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:50:15 smithi170 ceph-mon[27620]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-10T00:50:16.088 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:50:15 smithi106 ceph-mon[25742]: pgmap v114: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-03-10T00:50:16.089 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:50:15 smithi106 ceph-mon[25742]: from='osd.4 [v2:172.21.15.170:6800/335809953,v1:172.21.15.170:6801/335809953]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-10T00:50:16.089 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:50:15 smithi106 ceph-mon[25742]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-10T00:50:16.089 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:50:15 smithi106 ceph-mon[33460]: pgmap v114: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-03-10T00:50:16.089 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:50:15 smithi106 ceph-mon[33460]: from='osd.4 [v2:172.21.15.170:6800/335809953,v1:172.21.15.170:6801/335809953]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-10T00:50:16.089 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:50:15 smithi106 ceph-mon[33460]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-10T00:50:16.610 INFO:teuthology.orchestra.run.smithi170.stdout:Created osd(s) 4 on host 'smithi170' 2024-03-10T00:50:17.088 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:50:16 smithi106 ceph-mon[25742]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-03-10T00:50:17.089 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:50:16 smithi106 ceph-mon[25742]: from='osd.4 [v2:172.21.15.170:6800/335809953,v1:172.21.15.170:6801/335809953]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi170", "root=default"]}]: dispatch 2024-03-10T00:50:17.089 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:50:16 smithi106 ceph-mon[25742]: osdmap e28: 5 total, 4 up, 5 in 2024-03-10T00:50:17.089 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:50:16 smithi106 ceph-mon[25742]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-10T00:50:17.089 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:50:16 smithi106 ceph-mon[25742]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi170", "root=default"]}]: dispatch 2024-03-10T00:50:17.089 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:50:16 smithi106 ceph-mon[25742]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:50:17.089 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:50:16 smithi106 ceph-mon[25742]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-10T00:50:17.089 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:50:16 smithi106 ceph-mon[25742]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:50:17.089 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:50:16 smithi106 ceph-mon[25742]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-10T00:50:17.089 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:50:16 smithi106 ceph-mon[25742]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:50:17.089 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:50:16 smithi106 ceph-mon[25742]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-10T00:50:17.089 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:50:16 smithi106 ceph-mon[25742]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-10T00:50:17.090 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:50:16 smithi106 ceph-mon[25742]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:50:17.090 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:50:16 smithi106 ceph-mon[25742]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi170", "root=default"]}]': finished 2024-03-10T00:50:17.090 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:50:16 smithi106 ceph-mon[33460]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-03-10T00:50:17.090 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:50:16 smithi106 ceph-mon[33460]: from='osd.4 [v2:172.21.15.170:6800/335809953,v1:172.21.15.170:6801/335809953]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi170", "root=default"]}]: dispatch 2024-03-10T00:50:17.090 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:50:16 smithi106 ceph-mon[33460]: osdmap e28: 5 total, 4 up, 5 in 2024-03-10T00:50:17.090 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:50:16 smithi106 ceph-mon[33460]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-10T00:50:17.090 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:50:16 smithi106 ceph-mon[33460]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi170", "root=default"]}]: dispatch 2024-03-10T00:50:17.090 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:50:16 smithi106 ceph-mon[33460]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:50:17.090 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:50:16 smithi106 ceph-mon[33460]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-10T00:50:17.090 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:50:16 smithi106 ceph-mon[33460]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:50:17.091 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:50:16 smithi106 ceph-mon[33460]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-10T00:50:17.091 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:50:16 smithi106 ceph-mon[33460]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:50:17.091 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:50:16 smithi106 ceph-mon[33460]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-10T00:50:17.091 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:50:16 smithi106 ceph-mon[33460]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-10T00:50:17.091 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:50:16 smithi106 ceph-mon[33460]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:50:17.091 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:50:16 smithi106 ceph-mon[33460]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi170", "root=default"]}]': finished 2024-03-10T00:50:17.152 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:50:16 smithi170 ceph-mon[27620]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-03-10T00:50:17.153 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:50:16 smithi170 ceph-mon[27620]: from='osd.4 [v2:172.21.15.170:6800/335809953,v1:172.21.15.170:6801/335809953]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi170", "root=default"]}]: dispatch 2024-03-10T00:50:17.153 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:50:16 smithi170 ceph-mon[27620]: osdmap e28: 5 total, 4 up, 5 in 2024-03-10T00:50:17.153 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:50:16 smithi170 ceph-mon[27620]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-10T00:50:17.153 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:50:16 smithi170 ceph-mon[27620]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi170", "root=default"]}]: dispatch 2024-03-10T00:50:17.153 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:50:16 smithi170 ceph-mon[27620]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:50:17.153 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:50:16 smithi170 ceph-mon[27620]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-10T00:50:17.153 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:50:16 smithi170 ceph-mon[27620]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:50:17.153 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:50:16 smithi170 ceph-mon[27620]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-10T00:50:17.153 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:50:16 smithi170 ceph-mon[27620]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:50:17.153 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:50:16 smithi170 ceph-mon[27620]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-10T00:50:17.153 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:50:16 smithi170 ceph-mon[27620]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-10T00:50:17.153 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:50:16 smithi170 ceph-mon[27620]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:50:17.154 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:50:16 smithi170 ceph-mon[27620]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi170", "root=default"]}]': finished 2024-03-10T00:50:17.180 DEBUG:teuthology.orchestra.run.smithi170:osd.4> sudo journalctl -f -n 0 -u ceph-75c37498-de77-11ee-95c7-87774f69a715@osd.4.service 2024-03-10T00:50:17.183 INFO:tasks.cephadm:Deploying osd.5 on smithi170 with /dev/nvme3n1... 2024-03-10T00:50:17.183 DEBUG:teuthology.orchestra.run.smithi170:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:831bc188fa69f143bbba69ac76119e784500e8e3 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 75c37498-de77-11ee-95c7-87774f69a715 -- ceph-volume lvm zap /dev/nvme3n1 2024-03-10T00:50:17.455 INFO:teuthology.orchestra.run.smithi170.stderr:Inferring config /var/lib/ceph/75c37498-de77-11ee-95c7-87774f69a715/mon.b/config 2024-03-10T00:50:17.867 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:50:17 smithi106 ceph-mon[33460]: Adjusting osd_memory_target on smithi170 to 16064M 2024-03-10T00:50:17.867 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:50:17 smithi106 ceph-mon[33460]: pgmap v116: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-03-10T00:50:17.867 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:50:17 smithi106 ceph-mon[33460]: osdmap e29: 5 total, 4 up, 5 in 2024-03-10T00:50:17.867 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:50:17 smithi106 ceph-mon[33460]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-10T00:50:17.867 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:50:17 smithi106 ceph-mon[33460]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-10T00:50:17.868 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:50:17 smithi106 ceph-mon[25742]: Adjusting osd_memory_target on smithi170 to 16064M 2024-03-10T00:50:17.868 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:50:17 smithi106 ceph-mon[25742]: pgmap v116: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-03-10T00:50:17.868 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:50:17 smithi106 ceph-mon[25742]: osdmap e29: 5 total, 4 up, 5 in 2024-03-10T00:50:17.868 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:50:17 smithi106 ceph-mon[25742]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-10T00:50:17.868 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:50:17 smithi106 ceph-mon[25742]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-10T00:50:18.274 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:50:17 smithi170 ceph-mon[27620]: Adjusting osd_memory_target on smithi170 to 16064M 2024-03-10T00:50:18.274 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:50:17 smithi170 ceph-mon[27620]: pgmap v116: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-03-10T00:50:18.274 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:50:17 smithi170 ceph-mon[27620]: osdmap e29: 5 total, 4 up, 5 in 2024-03-10T00:50:18.275 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:50:17 smithi170 ceph-mon[27620]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-10T00:50:18.275 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:50:17 smithi170 ceph-mon[27620]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-10T00:50:19.083 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:50:18 smithi106 ceph-mon[25742]: purged_snaps scrub starts 2024-03-10T00:50:19.083 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:50:18 smithi106 ceph-mon[25742]: purged_snaps scrub ok 2024-03-10T00:50:19.083 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:50:18 smithi106 ceph-mon[25742]: osd.4 [v2:172.21.15.170:6800/335809953,v1:172.21.15.170:6801/335809953] boot 2024-03-10T00:50:19.083 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:50:18 smithi106 ceph-mon[25742]: osdmap e30: 5 total, 5 up, 5 in 2024-03-10T00:50:19.083 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:50:18 smithi106 ceph-mon[25742]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-10T00:50:19.083 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:50:18 smithi106 ceph-mon[33460]: purged_snaps scrub starts 2024-03-10T00:50:19.083 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:50:18 smithi106 ceph-mon[33460]: purged_snaps scrub ok 2024-03-10T00:50:19.083 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:50:18 smithi106 ceph-mon[33460]: osd.4 [v2:172.21.15.170:6800/335809953,v1:172.21.15.170:6801/335809953] boot 2024-03-10T00:50:19.083 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:50:18 smithi106 ceph-mon[33460]: osdmap e30: 5 total, 5 up, 5 in 2024-03-10T00:50:19.084 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:50:18 smithi106 ceph-mon[33460]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-10T00:50:19.274 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:50:18 smithi170 ceph-mon[27620]: purged_snaps scrub starts 2024-03-10T00:50:19.275 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:50:18 smithi170 ceph-mon[27620]: purged_snaps scrub ok 2024-03-10T00:50:19.275 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:50:18 smithi170 ceph-mon[27620]: osd.4 [v2:172.21.15.170:6800/335809953,v1:172.21.15.170:6801/335809953] boot 2024-03-10T00:50:19.275 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:50:18 smithi170 ceph-mon[27620]: osdmap e30: 5 total, 5 up, 5 in 2024-03-10T00:50:19.275 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:50:18 smithi170 ceph-mon[27620]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-10T00:50:19.389 INFO:teuthology.orchestra.run.smithi170.stderr:--> Zapping: /dev/nvme3n1 2024-03-10T00:50:19.390 INFO:teuthology.orchestra.run.smithi170.stderr:--> --destroy was not specified, but zapping a whole device will remove the partition table 2024-03-10T00:50:19.390 INFO:teuthology.orchestra.run.smithi170.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/nvme3n1 bs=1M count=10 conv=fsync 2024-03-10T00:50:19.390 INFO:teuthology.orchestra.run.smithi170.stderr: stderr: 10+0 records in 2024-03-10T00:50:19.390 INFO:teuthology.orchestra.run.smithi170.stderr:10+0 records out 2024-03-10T00:50:19.390 INFO:teuthology.orchestra.run.smithi170.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.00989213 s, 1.1 GB/s 2024-03-10T00:50:19.390 INFO:teuthology.orchestra.run.smithi170.stderr:--> Zapping successful for: 2024-03-10T00:50:20.024 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:50:19 smithi170 ceph-mon[27620]: pgmap v119: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-03-10T00:50:20.024 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:50:19 smithi170 ceph-mon[27620]: osdmap e31: 5 total, 5 up, 5 in 2024-03-10T00:50:20.024 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:50:19 smithi170 ceph-mon[27620]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:50:20.025 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:50:19 smithi170 ceph-mon[27620]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:50:20.025 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:50:19 smithi170 ceph-mon[27620]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:50:20.025 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:50:19 smithi170 ceph-mon[27620]: osdmap e32: 5 total, 5 up, 5 in 2024-03-10T00:50:20.085 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:50:19 smithi106 ceph-mon[25742]: pgmap v119: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-03-10T00:50:20.085 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:50:19 smithi106 ceph-mon[25742]: osdmap e31: 5 total, 5 up, 5 in 2024-03-10T00:50:20.085 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:50:19 smithi106 ceph-mon[25742]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:50:20.085 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:50:19 smithi106 ceph-mon[25742]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:50:20.085 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:50:19 smithi106 ceph-mon[25742]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:50:20.086 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:50:19 smithi106 ceph-mon[25742]: osdmap e32: 5 total, 5 up, 5 in 2024-03-10T00:50:20.086 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:50:19 smithi106 ceph-mon[33460]: pgmap v119: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-03-10T00:50:20.086 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:50:19 smithi106 ceph-mon[33460]: osdmap e31: 5 total, 5 up, 5 in 2024-03-10T00:50:20.086 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:50:19 smithi106 ceph-mon[33460]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:50:20.086 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:50:19 smithi106 ceph-mon[33460]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:50:20.087 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:50:19 smithi106 ceph-mon[33460]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:50:20.087 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:50:19 smithi106 ceph-mon[33460]: osdmap e32: 5 total, 5 up, 5 in 2024-03-10T00:50:20.101 DEBUG:teuthology.orchestra.run.smithi170:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:831bc188fa69f143bbba69ac76119e784500e8e3 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 75c37498-de77-11ee-95c7-87774f69a715 -- ceph orch daemon add osd smithi170:/dev/nvme3n1 2024-03-10T00:50:20.334 INFO:teuthology.orchestra.run.smithi170.stderr:Inferring config /var/lib/ceph/75c37498-de77-11ee-95c7-87774f69a715/mon.b/config 2024-03-10T00:50:21.396 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:50:21 smithi170 ceph-mon[27620]: osdmap e33: 5 total, 5 up, 5 in 2024-03-10T00:50:21.396 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:50:21 smithi170 ceph-mon[27620]: pgmap v123: 1 pgs: 1 remapped+peering; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-03-10T00:50:21.588 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:50:21 smithi106 ceph-mon[25742]: osdmap e33: 5 total, 5 up, 5 in 2024-03-10T00:50:21.589 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:50:21 smithi106 ceph-mon[25742]: pgmap v123: 1 pgs: 1 remapped+peering; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-03-10T00:50:21.589 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:50:21 smithi106 ceph-mon[33460]: osdmap e33: 5 total, 5 up, 5 in 2024-03-10T00:50:21.589 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:50:21 smithi106 ceph-mon[33460]: pgmap v123: 1 pgs: 1 remapped+peering; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-03-10T00:50:22.524 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:50:22 smithi170 ceph-mon[27620]: osdmap e34: 5 total, 5 up, 5 in 2024-03-10T00:50:22.524 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:50:22 smithi170 ceph-mon[27620]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-10T00:50:22.524 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:50:22 smithi170 ceph-mon[27620]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-10T00:50:22.524 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:50:22 smithi170 ceph-mon[27620]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-10T00:50:22.588 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:50:22 smithi106 ceph-mon[25742]: osdmap e34: 5 total, 5 up, 5 in 2024-03-10T00:50:22.589 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:50:22 smithi106 ceph-mon[25742]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-10T00:50:22.589 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:50:22 smithi106 ceph-mon[25742]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-10T00:50:22.589 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:50:22 smithi106 ceph-mon[25742]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-10T00:50:22.589 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:50:22 smithi106 ceph-mon[33460]: osdmap e34: 5 total, 5 up, 5 in 2024-03-10T00:50:22.589 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:50:22 smithi106 ceph-mon[33460]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-10T00:50:22.589 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:50:22 smithi106 ceph-mon[33460]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-10T00:50:22.589 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:50:22 smithi106 ceph-mon[33460]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-10T00:50:23.274 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:50:23 smithi170 ceph-mon[27620]: from='client.24214 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi170:/dev/nvme3n1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-10T00:50:23.274 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:50:23 smithi170 ceph-mon[27620]: pgmap v125: 1 pgs: 1 remapped+peering; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-03-10T00:50:23.590 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:50:23 smithi106 ceph-mon[25742]: from='client.24214 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi170:/dev/nvme3n1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-10T00:50:23.590 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:50:23 smithi106 ceph-mon[25742]: pgmap v125: 1 pgs: 1 remapped+peering; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-03-10T00:50:23.590 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:50:23 smithi106 ceph-mon[33460]: from='client.24214 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi170:/dev/nvme3n1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-10T00:50:23.590 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:50:23 smithi106 ceph-mon[33460]: pgmap v125: 1 pgs: 1 remapped+peering; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-03-10T00:50:25.088 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:50:24 smithi106 ceph-mon[25742]: from='client.? 172.21.15.170:0/2304854915' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "cf14b75a-d753-4279-9b67-593fed3c5785"}]: dispatch 2024-03-10T00:50:25.089 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:50:24 smithi106 ceph-mon[25742]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "cf14b75a-d753-4279-9b67-593fed3c5785"}]: dispatch 2024-03-10T00:50:25.089 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:50:24 smithi106 ceph-mon[25742]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "cf14b75a-d753-4279-9b67-593fed3c5785"}]': finished 2024-03-10T00:50:25.089 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:50:24 smithi106 ceph-mon[25742]: osdmap e35: 6 total, 5 up, 6 in 2024-03-10T00:50:25.089 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:50:24 smithi106 ceph-mon[25742]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-10T00:50:25.089 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:50:24 smithi106 ceph-mon[33460]: from='client.? 172.21.15.170:0/2304854915' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "cf14b75a-d753-4279-9b67-593fed3c5785"}]: dispatch 2024-03-10T00:50:25.089 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:50:24 smithi106 ceph-mon[33460]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "cf14b75a-d753-4279-9b67-593fed3c5785"}]: dispatch 2024-03-10T00:50:25.089 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:50:24 smithi106 ceph-mon[33460]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "cf14b75a-d753-4279-9b67-593fed3c5785"}]': finished 2024-03-10T00:50:25.089 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:50:24 smithi106 ceph-mon[33460]: osdmap e35: 6 total, 5 up, 6 in 2024-03-10T00:50:25.089 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:50:24 smithi106 ceph-mon[33460]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-10T00:50:25.274 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:50:24 smithi170 ceph-mon[27620]: from='client.? 172.21.15.170:0/2304854915' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "cf14b75a-d753-4279-9b67-593fed3c5785"}]: dispatch 2024-03-10T00:50:25.274 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:50:24 smithi170 ceph-mon[27620]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "cf14b75a-d753-4279-9b67-593fed3c5785"}]: dispatch 2024-03-10T00:50:25.275 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:50:24 smithi170 ceph-mon[27620]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "cf14b75a-d753-4279-9b67-593fed3c5785"}]': finished 2024-03-10T00:50:25.275 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:50:24 smithi170 ceph-mon[27620]: osdmap e35: 6 total, 5 up, 6 in 2024-03-10T00:50:25.275 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:50:24 smithi170 ceph-mon[27620]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-10T00:50:26.088 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:50:25 smithi106 ceph-mon[25742]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 144 MiB used, 447 GiB / 447 GiB avail; 97 KiB/s, 0 objects/s recovering 2024-03-10T00:50:26.089 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:50:25 smithi106 ceph-mon[33460]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 144 MiB used, 447 GiB / 447 GiB avail; 97 KiB/s, 0 objects/s recovering 2024-03-10T00:50:26.274 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:50:25 smithi170 ceph-mon[27620]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 144 MiB used, 447 GiB / 447 GiB avail; 97 KiB/s, 0 objects/s recovering 2024-03-10T00:50:27.088 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:50:26 smithi106 ceph-mon[25742]: from='client.? 172.21.15.170:0/2230601939' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-10T00:50:27.089 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:50:26 smithi106 ceph-mon[33460]: from='client.? 172.21.15.170:0/2230601939' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-10T00:50:27.148 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:50:26 smithi170 ceph-mon[27620]: from='client.? 172.21.15.170:0/2230601939' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-10T00:50:28.088 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:50:27 smithi106 ceph-mon[33460]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail; 87 KiB/s, 0 objects/s recovering 2024-03-10T00:50:28.089 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:50:27 smithi106 ceph-mon[33460]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:50:28.089 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:50:27 smithi106 ceph-mon[33460]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:50:28.089 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:50:27 smithi106 ceph-mon[33460]: Detected new or changed devices on smithi170 2024-03-10T00:50:28.089 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:50:27 smithi106 ceph-mon[33460]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:50:28.089 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:50:27 smithi106 ceph-mon[33460]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-10T00:50:28.089 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:50:27 smithi106 ceph-mon[33460]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-10T00:50:28.089 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:50:27 smithi106 ceph-mon[33460]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-10T00:50:28.089 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:50:27 smithi106 ceph-mon[33460]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail; 94 KiB/s, 0 objects/s recovering 2024-03-10T00:50:28.089 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:50:27 smithi106 ceph-mon[33460]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:50:28.089 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:50:27 smithi106 ceph-mon[33460]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:50:28.089 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:50:27 smithi106 ceph-mon[33460]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:50:28.089 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:50:27 smithi106 ceph-mon[33460]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:50:28.089 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:50:27 smithi106 ceph-mon[33460]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-10T00:50:28.089 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:50:27 smithi106 ceph-mon[33460]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-10T00:50:28.090 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:50:27 smithi106 ceph-mon[33460]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-10T00:50:28.090 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:50:27 smithi106 ceph-mon[33460]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:50:28.090 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:50:27 smithi106 ceph-mon[25742]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail; 87 KiB/s, 0 objects/s recovering 2024-03-10T00:50:28.090 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:50:27 smithi106 ceph-mon[25742]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:50:28.090 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:50:27 smithi106 ceph-mon[25742]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:50:28.090 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:50:27 smithi106 ceph-mon[25742]: Detected new or changed devices on smithi170 2024-03-10T00:50:28.090 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:50:27 smithi106 ceph-mon[25742]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:50:28.090 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:50:27 smithi106 ceph-mon[25742]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-10T00:50:28.090 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:50:27 smithi106 ceph-mon[25742]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-10T00:50:28.090 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:50:27 smithi106 ceph-mon[25742]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-10T00:50:28.090 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:50:27 smithi106 ceph-mon[25742]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail; 94 KiB/s, 0 objects/s recovering 2024-03-10T00:50:28.091 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:50:27 smithi106 ceph-mon[25742]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:50:28.091 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:50:27 smithi106 ceph-mon[25742]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:50:28.091 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:50:27 smithi106 ceph-mon[25742]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:50:28.091 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:50:27 smithi106 ceph-mon[25742]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:50:28.091 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:50:27 smithi106 ceph-mon[25742]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-10T00:50:28.091 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:50:27 smithi106 ceph-mon[25742]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-10T00:50:28.091 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:50:27 smithi106 ceph-mon[25742]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-10T00:50:28.091 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:50:27 smithi106 ceph-mon[25742]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:50:28.274 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:50:27 smithi170 ceph-mon[27620]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail; 87 KiB/s, 0 objects/s recovering 2024-03-10T00:50:28.275 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:50:27 smithi170 ceph-mon[27620]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:50:28.275 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:50:27 smithi170 ceph-mon[27620]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:50:28.275 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:50:27 smithi170 ceph-mon[27620]: Detected new or changed devices on smithi170 2024-03-10T00:50:28.275 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:50:27 smithi170 ceph-mon[27620]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:50:28.275 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:50:27 smithi170 ceph-mon[27620]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-10T00:50:28.275 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:50:27 smithi170 ceph-mon[27620]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-10T00:50:28.275 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:50:27 smithi170 ceph-mon[27620]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-10T00:50:28.275 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:50:27 smithi170 ceph-mon[27620]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail; 94 KiB/s, 0 objects/s recovering 2024-03-10T00:50:28.275 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:50:27 smithi170 ceph-mon[27620]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:50:28.275 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:50:27 smithi170 ceph-mon[27620]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:50:28.275 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:50:27 smithi170 ceph-mon[27620]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:50:28.275 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:50:27 smithi170 ceph-mon[27620]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:50:28.275 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:50:27 smithi170 ceph-mon[27620]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-10T00:50:28.276 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:50:27 smithi170 ceph-mon[27620]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-10T00:50:28.276 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:50:27 smithi170 ceph-mon[27620]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-10T00:50:28.276 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:50:27 smithi170 ceph-mon[27620]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:50:29.274 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:50:28 smithi170 ceph-mon[27620]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-03-10T00:50:29.338 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:50:28 smithi106 ceph-mon[25742]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-03-10T00:50:29.339 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:50:28 smithi106 ceph-mon[33460]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-03-10T00:50:31.267 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:50:30 smithi170 ceph-mon[27620]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail; 90 KiB/s, 0 objects/s recovering 2024-03-10T00:50:31.338 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:50:30 smithi106 ceph-mon[25742]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail; 90 KiB/s, 0 objects/s recovering 2024-03-10T00:50:31.339 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:50:30 smithi106 ceph-mon[33460]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail; 90 KiB/s, 0 objects/s recovering 2024-03-10T00:50:33.032 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:50:32 smithi170 ceph-mon[27620]: pgmap v132: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-03-10T00:50:33.032 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:50:32 smithi170 ceph-mon[27620]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-03-10T00:50:33.032 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:50:32 smithi170 ceph-mon[27620]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-10T00:50:33.338 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:50:32 smithi106 ceph-mon[33460]: pgmap v132: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-03-10T00:50:33.338 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:50:32 smithi106 ceph-mon[33460]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-03-10T00:50:33.338 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:50:32 smithi106 ceph-mon[33460]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-10T00:50:33.339 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:50:32 smithi106 ceph-mon[25742]: pgmap v132: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-03-10T00:50:33.339 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:50:32 smithi106 ceph-mon[25742]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-03-10T00:50:33.339 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:50:32 smithi106 ceph-mon[25742]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-10T00:50:34.113 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:50:33 smithi170 ceph-mon[27620]: Deploying daemon osd.5 on smithi170 2024-03-10T00:50:34.338 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:50:33 smithi106 ceph-mon[25742]: Deploying daemon osd.5 on smithi170 2024-03-10T00:50:34.338 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:50:33 smithi106 ceph-mon[33460]: Deploying daemon osd.5 on smithi170 2024-03-10T00:50:35.102 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:50:34 smithi170 ceph-mon[27620]: pgmap v133: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-03-10T00:50:35.338 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:50:34 smithi106 ceph-mon[25742]: pgmap v133: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-03-10T00:50:35.339 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:50:34 smithi106 ceph-mon[33460]: pgmap v133: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-03-10T00:50:37.167 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:50:36 smithi170 ceph-mon[27620]: pgmap v134: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-03-10T00:50:37.338 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:50:36 smithi106 ceph-mon[25742]: pgmap v134: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-03-10T00:50:37.339 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:50:36 smithi106 ceph-mon[33460]: pgmap v134: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-03-10T00:50:39.024 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:50:38 smithi170 ceph-mon[27620]: pgmap v135: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-03-10T00:50:39.024 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:50:38 smithi170 ceph-mon[27620]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:50:39.025 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:50:38 smithi170 ceph-mon[27620]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:50:39.025 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:50:38 smithi170 ceph-mon[27620]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-10T00:50:39.025 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:50:38 smithi170 ceph-mon[27620]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-10T00:50:39.025 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:50:38 smithi170 ceph-mon[27620]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-10T00:50:39.025 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:50:38 smithi170 ceph-mon[27620]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:50:39.088 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:50:38 smithi106 ceph-mon[25742]: pgmap v135: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-03-10T00:50:39.088 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:50:38 smithi106 ceph-mon[25742]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:50:39.089 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:50:38 smithi106 ceph-mon[25742]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:50:39.089 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:50:38 smithi106 ceph-mon[25742]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-10T00:50:39.089 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:50:38 smithi106 ceph-mon[25742]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-10T00:50:39.089 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:50:38 smithi106 ceph-mon[25742]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-10T00:50:39.089 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:50:38 smithi106 ceph-mon[25742]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:50:39.089 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:50:38 smithi106 ceph-mon[33460]: pgmap v135: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-03-10T00:50:39.089 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:50:38 smithi106 ceph-mon[33460]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:50:39.089 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:50:38 smithi106 ceph-mon[33460]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:50:39.089 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:50:38 smithi106 ceph-mon[33460]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-10T00:50:39.089 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:50:38 smithi106 ceph-mon[33460]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-10T00:50:39.090 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:50:38 smithi106 ceph-mon[33460]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-10T00:50:39.090 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:50:38 smithi106 ceph-mon[33460]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:50:40.571 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:50:40 smithi170 ceph-mon[27620]: pgmap v136: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-03-10T00:50:40.571 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:50:40 smithi170 ceph-mon[27620]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:50:40.571 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:50:40 smithi170 ceph-mon[27620]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:50:40.571 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:50:40 smithi170 ceph-mon[27620]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:50:40.838 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:50:40 smithi106 ceph-mon[25742]: pgmap v136: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-03-10T00:50:40.839 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:50:40 smithi106 ceph-mon[25742]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:50:40.839 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:50:40 smithi106 ceph-mon[25742]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:50:40.839 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:50:40 smithi106 ceph-mon[25742]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:50:40.839 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:50:40 smithi106 ceph-mon[33460]: pgmap v136: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-03-10T00:50:40.839 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:50:40 smithi106 ceph-mon[33460]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:50:40.839 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:50:40 smithi106 ceph-mon[33460]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:50:40.839 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:50:40 smithi106 ceph-mon[33460]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:50:41.774 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:50:41 smithi170 ceph-mon[27620]: from='osd.5 [v2:172.21.15.170:6808/47584024,v1:172.21.15.170:6809/47584024]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-10T00:50:41.774 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:50:41 smithi170 ceph-mon[27620]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-10T00:50:41.838 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:50:41 smithi106 ceph-mon[25742]: from='osd.5 [v2:172.21.15.170:6808/47584024,v1:172.21.15.170:6809/47584024]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-10T00:50:41.838 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:50:41 smithi106 ceph-mon[25742]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-10T00:50:41.839 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:50:41 smithi106 ceph-mon[33460]: from='osd.5 [v2:172.21.15.170:6808/47584024,v1:172.21.15.170:6809/47584024]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-10T00:50:41.839 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:50:41 smithi106 ceph-mon[33460]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-10T00:50:41.867 INFO:teuthology.orchestra.run.smithi170.stdout:Created osd(s) 5 on host 'smithi170' 2024-03-10T00:50:42.524 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:50:42 smithi170 ceph-mon[27620]: pgmap v137: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-03-10T00:50:42.524 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:50:42 smithi170 ceph-mon[27620]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-03-10T00:50:42.525 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:50:42 smithi170 ceph-mon[27620]: from='osd.5 [v2:172.21.15.170:6808/47584024,v1:172.21.15.170:6809/47584024]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi170", "root=default"]}]: dispatch 2024-03-10T00:50:42.525 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:50:42 smithi170 ceph-mon[27620]: osdmap e36: 6 total, 5 up, 6 in 2024-03-10T00:50:42.525 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:50:42 smithi170 ceph-mon[27620]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-10T00:50:42.525 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:50:42 smithi170 ceph-mon[27620]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi170", "root=default"]}]: dispatch 2024-03-10T00:50:42.525 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:50:42 smithi170 ceph-mon[27620]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:50:42.525 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:50:42 smithi170 ceph-mon[27620]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:50:42.525 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:50:42 smithi170 ceph-mon[27620]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-10T00:50:42.525 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:50:42 smithi170 ceph-mon[27620]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-10T00:50:42.525 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:50:42 smithi170 ceph-mon[27620]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-10T00:50:42.525 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:50:42 smithi170 ceph-mon[27620]: Adjusting osd_memory_target on smithi170 to 8032M 2024-03-10T00:50:42.525 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:50:42 smithi170 ceph-mon[27620]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:50:42.525 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:50:42 smithi170 ceph-mon[27620]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-10T00:50:42.525 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:50:42 smithi170 ceph-mon[27620]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-10T00:50:42.525 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:50:42 smithi170 ceph-mon[27620]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:50:42.681 DEBUG:teuthology.orchestra.run.smithi170:osd.5> sudo journalctl -f -n 0 -u ceph-75c37498-de77-11ee-95c7-87774f69a715@osd.5.service 2024-03-10T00:50:42.683 INFO:tasks.cephadm:Deploying osd.6 on smithi170 with /dev/nvme2n1... 2024-03-10T00:50:42.683 DEBUG:teuthology.orchestra.run.smithi170:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:831bc188fa69f143bbba69ac76119e784500e8e3 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 75c37498-de77-11ee-95c7-87774f69a715 -- ceph-volume lvm zap /dev/nvme2n1 2024-03-10T00:50:42.839 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:50:42 smithi106 ceph-mon[25742]: pgmap v137: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-03-10T00:50:42.839 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:50:42 smithi106 ceph-mon[25742]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-03-10T00:50:42.839 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:50:42 smithi106 ceph-mon[25742]: from='osd.5 [v2:172.21.15.170:6808/47584024,v1:172.21.15.170:6809/47584024]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi170", "root=default"]}]: dispatch 2024-03-10T00:50:42.839 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:50:42 smithi106 ceph-mon[25742]: osdmap e36: 6 total, 5 up, 6 in 2024-03-10T00:50:42.839 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:50:42 smithi106 ceph-mon[25742]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-10T00:50:42.839 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:50:42 smithi106 ceph-mon[25742]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi170", "root=default"]}]: dispatch 2024-03-10T00:50:42.839 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:50:42 smithi106 ceph-mon[25742]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:50:42.839 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:50:42 smithi106 ceph-mon[25742]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:50:42.839 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:50:42 smithi106 ceph-mon[25742]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-10T00:50:42.839 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:50:42 smithi106 ceph-mon[25742]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-10T00:50:42.839 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:50:42 smithi106 ceph-mon[25742]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-10T00:50:42.840 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:50:42 smithi106 ceph-mon[25742]: Adjusting osd_memory_target on smithi170 to 8032M 2024-03-10T00:50:42.840 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:50:42 smithi106 ceph-mon[25742]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:50:42.840 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:50:42 smithi106 ceph-mon[25742]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-10T00:50:42.840 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:50:42 smithi106 ceph-mon[25742]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-10T00:50:42.840 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:50:42 smithi106 ceph-mon[25742]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:50:42.840 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:50:42 smithi106 ceph-mon[33460]: pgmap v137: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-03-10T00:50:42.840 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:50:42 smithi106 ceph-mon[33460]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-03-10T00:50:42.840 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:50:42 smithi106 ceph-mon[33460]: from='osd.5 [v2:172.21.15.170:6808/47584024,v1:172.21.15.170:6809/47584024]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi170", "root=default"]}]: dispatch 2024-03-10T00:50:42.840 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:50:42 smithi106 ceph-mon[33460]: osdmap e36: 6 total, 5 up, 6 in 2024-03-10T00:50:42.840 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:50:42 smithi106 ceph-mon[33460]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-10T00:50:42.840 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:50:42 smithi106 ceph-mon[33460]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi170", "root=default"]}]: dispatch 2024-03-10T00:50:42.841 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:50:42 smithi106 ceph-mon[33460]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:50:42.841 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:50:42 smithi106 ceph-mon[33460]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:50:42.841 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:50:42 smithi106 ceph-mon[33460]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-10T00:50:42.841 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:50:42 smithi106 ceph-mon[33460]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-10T00:50:42.841 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:50:42 smithi106 ceph-mon[33460]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-10T00:50:42.841 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:50:42 smithi106 ceph-mon[33460]: Adjusting osd_memory_target on smithi170 to 8032M 2024-03-10T00:50:42.841 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:50:42 smithi106 ceph-mon[33460]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:50:42.841 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:50:42 smithi106 ceph-mon[33460]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-10T00:50:42.841 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:50:42 smithi106 ceph-mon[33460]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-10T00:50:42.841 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:50:42 smithi106 ceph-mon[33460]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:50:42.959 INFO:teuthology.orchestra.run.smithi170.stderr:Inferring config /var/lib/ceph/75c37498-de77-11ee-95c7-87774f69a715/mon.b/config 2024-03-10T00:50:43.024 INFO:journalctl@ceph.osd.5.smithi170.stdout:Mar 10 00:50:42 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-osd-5[38986]: 2024-03-10T00:50:42.730+0000 7f724aefb700 -1 osd.5 0 waiting for initial osdmap 2024-03-10T00:50:43.024 INFO:journalctl@ceph.osd.5.smithi170.stdout:Mar 10 00:50:42 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-osd-5[38986]: 2024-03-10T00:50:42.735+0000 7f7242ce6700 -1 osd.5 37 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-10T00:50:43.774 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:50:43 smithi170 ceph-mon[27620]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi170", "root=default"]}]': finished 2024-03-10T00:50:43.774 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:50:43 smithi170 ceph-mon[27620]: osdmap e37: 6 total, 5 up, 6 in 2024-03-10T00:50:43.775 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:50:43 smithi170 ceph-mon[27620]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-10T00:50:43.775 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:50:43 smithi170 ceph-mon[27620]: from='osd.5 ' entity='osd.5' 2024-03-10T00:50:43.775 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:50:43 smithi170 ceph-mon[27620]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-10T00:50:43.838 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:50:43 smithi106 ceph-mon[25742]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi170", "root=default"]}]': finished 2024-03-10T00:50:43.839 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:50:43 smithi106 ceph-mon[25742]: osdmap e37: 6 total, 5 up, 6 in 2024-03-10T00:50:43.839 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:50:43 smithi106 ceph-mon[25742]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-10T00:50:43.839 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:50:43 smithi106 ceph-mon[25742]: from='osd.5 ' entity='osd.5' 2024-03-10T00:50:43.839 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:50:43 smithi106 ceph-mon[25742]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-10T00:50:43.839 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:50:43 smithi106 ceph-mon[33460]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi170", "root=default"]}]': finished 2024-03-10T00:50:43.839 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:50:43 smithi106 ceph-mon[33460]: osdmap e37: 6 total, 5 up, 6 in 2024-03-10T00:50:43.839 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:50:43 smithi106 ceph-mon[33460]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-10T00:50:43.839 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:50:43 smithi106 ceph-mon[33460]: from='osd.5 ' entity='osd.5' 2024-03-10T00:50:43.840 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:50:43 smithi106 ceph-mon[33460]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-10T00:50:44.553 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:50:44 smithi170 ceph-mon[27620]: purged_snaps scrub starts 2024-03-10T00:50:44.553 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:50:44 smithi170 ceph-mon[27620]: purged_snaps scrub ok 2024-03-10T00:50:44.554 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:50:44 smithi170 ceph-mon[27620]: pgmap v140: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-03-10T00:50:44.554 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:50:44 smithi170 ceph-mon[27620]: osd.5 [v2:172.21.15.170:6808/47584024,v1:172.21.15.170:6809/47584024] boot 2024-03-10T00:50:44.554 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:50:44 smithi170 ceph-mon[27620]: osdmap e38: 6 total, 6 up, 6 in 2024-03-10T00:50:44.554 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:50:44 smithi170 ceph-mon[27620]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-10T00:50:44.554 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:50:44 smithi170 ceph-mon[27620]: osdmap e39: 6 total, 6 up, 6 in 2024-03-10T00:50:44.838 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:50:44 smithi106 ceph-mon[25742]: purged_snaps scrub starts 2024-03-10T00:50:44.839 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:50:44 smithi106 ceph-mon[25742]: purged_snaps scrub ok 2024-03-10T00:50:44.839 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:50:44 smithi106 ceph-mon[25742]: pgmap v140: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-03-10T00:50:44.839 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:50:44 smithi106 ceph-mon[25742]: osd.5 [v2:172.21.15.170:6808/47584024,v1:172.21.15.170:6809/47584024] boot 2024-03-10T00:50:44.839 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:50:44 smithi106 ceph-mon[25742]: osdmap e38: 6 total, 6 up, 6 in 2024-03-10T00:50:44.839 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:50:44 smithi106 ceph-mon[25742]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-10T00:50:44.839 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:50:44 smithi106 ceph-mon[25742]: osdmap e39: 6 total, 6 up, 6 in 2024-03-10T00:50:44.839 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:50:44 smithi106 ceph-mon[33460]: purged_snaps scrub starts 2024-03-10T00:50:44.839 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:50:44 smithi106 ceph-mon[33460]: purged_snaps scrub ok 2024-03-10T00:50:44.839 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:50:44 smithi106 ceph-mon[33460]: pgmap v140: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-03-10T00:50:44.840 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:50:44 smithi106 ceph-mon[33460]: osd.5 [v2:172.21.15.170:6808/47584024,v1:172.21.15.170:6809/47584024] boot 2024-03-10T00:50:44.840 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:50:44 smithi106 ceph-mon[33460]: osdmap e38: 6 total, 6 up, 6 in 2024-03-10T00:50:44.840 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:50:44 smithi106 ceph-mon[33460]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-10T00:50:44.840 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:50:44 smithi106 ceph-mon[33460]: osdmap e39: 6 total, 6 up, 6 in 2024-03-10T00:50:44.907 INFO:teuthology.orchestra.run.smithi170.stderr:--> Zapping: /dev/nvme2n1 2024-03-10T00:50:44.907 INFO:teuthology.orchestra.run.smithi170.stderr:--> --destroy was not specified, but zapping a whole device will remove the partition table 2024-03-10T00:50:44.908 INFO:teuthology.orchestra.run.smithi170.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/nvme2n1 bs=1M count=10 conv=fsync 2024-03-10T00:50:44.908 INFO:teuthology.orchestra.run.smithi170.stderr: stderr: 10+0 records in 2024-03-10T00:50:44.908 INFO:teuthology.orchestra.run.smithi170.stderr:10+0 records out 2024-03-10T00:50:44.908 INFO:teuthology.orchestra.run.smithi170.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0107505 s, 975 MB/s 2024-03-10T00:50:44.908 INFO:teuthology.orchestra.run.smithi170.stderr:--> Zapping successful for: 2024-03-10T00:50:45.559 DEBUG:teuthology.orchestra.run.smithi170:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:831bc188fa69f143bbba69ac76119e784500e8e3 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 75c37498-de77-11ee-95c7-87774f69a715 -- ceph orch daemon add osd smithi170:/dev/nvme2n1 2024-03-10T00:50:45.794 INFO:teuthology.orchestra.run.smithi170.stderr:Inferring config /var/lib/ceph/75c37498-de77-11ee-95c7-87774f69a715/mon.b/config 2024-03-10T00:50:46.526 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:50:46 smithi170 ceph-mon[27620]: pgmap v143: 1 pgs: 1 unknown; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-03-10T00:50:46.526 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:50:46 smithi170 ceph-mon[27620]: osdmap e40: 6 total, 6 up, 6 in 2024-03-10T00:50:46.838 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:50:46 smithi106 ceph-mon[25742]: pgmap v143: 1 pgs: 1 unknown; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-03-10T00:50:46.838 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:50:46 smithi106 ceph-mon[25742]: osdmap e40: 6 total, 6 up, 6 in 2024-03-10T00:50:46.839 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:50:46 smithi106 ceph-mon[33460]: pgmap v143: 1 pgs: 1 unknown; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-03-10T00:50:46.839 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:50:46 smithi106 ceph-mon[33460]: osdmap e40: 6 total, 6 up, 6 in 2024-03-10T00:50:47.774 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:50:47 smithi170 ceph-mon[27620]: osdmap e41: 6 total, 6 up, 6 in 2024-03-10T00:50:47.774 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:50:47 smithi170 ceph-mon[27620]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:50:47.774 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:50:47 smithi170 ceph-mon[27620]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:50:47.775 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:50:47 smithi170 ceph-mon[27620]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:50:47.775 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:50:47 smithi170 ceph-mon[27620]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-10T00:50:47.775 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:50:47 smithi170 ceph-mon[27620]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-10T00:50:47.775 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:50:47 smithi170 ceph-mon[27620]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-10T00:50:47.775 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:50:47 smithi170 ceph-mon[27620]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:50:47.819 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:50:47 smithi106 ceph-mon[33460]: osdmap e41: 6 total, 6 up, 6 in 2024-03-10T00:50:47.819 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:50:47 smithi106 ceph-mon[33460]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:50:47.819 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:50:47 smithi106 ceph-mon[33460]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:50:47.819 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:50:47 smithi106 ceph-mon[33460]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:50:47.820 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:50:47 smithi106 ceph-mon[33460]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-10T00:50:47.820 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:50:47 smithi106 ceph-mon[33460]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-10T00:50:47.820 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:50:47 smithi106 ceph-mon[33460]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-10T00:50:47.820 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:50:47 smithi106 ceph-mon[33460]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:50:47.820 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:50:47 smithi106 ceph-mon[25742]: osdmap e41: 6 total, 6 up, 6 in 2024-03-10T00:50:47.820 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:50:47 smithi106 ceph-mon[25742]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:50:47.820 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:50:47 smithi106 ceph-mon[25742]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:50:47.820 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:50:47 smithi106 ceph-mon[25742]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:50:47.820 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:50:47 smithi106 ceph-mon[25742]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-10T00:50:47.820 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:50:47 smithi106 ceph-mon[25742]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-10T00:50:47.820 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:50:47 smithi106 ceph-mon[25742]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-10T00:50:47.820 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:50:47 smithi106 ceph-mon[25742]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:50:48.774 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:50:48 smithi170 ceph-mon[27620]: pgmap v146: 1 pgs: 1 unknown; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-03-10T00:50:48.774 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:50:48 smithi170 ceph-mon[27620]: Detected new or changed devices on smithi170 2024-03-10T00:50:48.774 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:50:48 smithi170 ceph-mon[27620]: pgmap v147: 1 pgs: 1 unknown; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-03-10T00:50:48.774 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:50:48 smithi170 ceph-mon[27620]: Health check failed: 1 stray daemon(s) not managed by cephadm (CEPHADM_STRAY_DAEMON) 2024-03-10T00:50:48.774 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:50:48 smithi170 ceph-mon[27620]: osdmap e42: 6 total, 6 up, 6 in 2024-03-10T00:50:48.774 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:50:48 smithi170 ceph-mon[27620]: from='client.24241 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi170:/dev/nvme2n1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-10T00:50:48.774 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:50:48 smithi170 ceph-mon[27620]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-10T00:50:48.774 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:50:48 smithi170 ceph-mon[27620]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-10T00:50:48.774 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:50:48 smithi170 ceph-mon[27620]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-10T00:50:48.838 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:50:48 smithi106 ceph-mon[25742]: pgmap v146: 1 pgs: 1 unknown; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-03-10T00:50:48.839 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:50:48 smithi106 ceph-mon[25742]: Detected new or changed devices on smithi170 2024-03-10T00:50:48.839 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:50:48 smithi106 ceph-mon[25742]: pgmap v147: 1 pgs: 1 unknown; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-03-10T00:50:48.839 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:50:48 smithi106 ceph-mon[25742]: Health check failed: 1 stray daemon(s) not managed by cephadm (CEPHADM_STRAY_DAEMON) 2024-03-10T00:50:48.839 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:50:48 smithi106 ceph-mon[25742]: osdmap e42: 6 total, 6 up, 6 in 2024-03-10T00:50:48.839 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:50:48 smithi106 ceph-mon[25742]: from='client.24241 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi170:/dev/nvme2n1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-10T00:50:48.839 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:50:48 smithi106 ceph-mon[25742]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-10T00:50:48.839 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:50:48 smithi106 ceph-mon[25742]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-10T00:50:48.839 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:50:48 smithi106 ceph-mon[25742]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-10T00:50:48.839 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:50:48 smithi106 ceph-mon[33460]: pgmap v146: 1 pgs: 1 unknown; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-03-10T00:50:48.840 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:50:48 smithi106 ceph-mon[33460]: Detected new or changed devices on smithi170 2024-03-10T00:50:48.840 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:50:48 smithi106 ceph-mon[33460]: pgmap v147: 1 pgs: 1 unknown; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-03-10T00:50:48.840 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:50:48 smithi106 ceph-mon[33460]: Health check failed: 1 stray daemon(s) not managed by cephadm (CEPHADM_STRAY_DAEMON) 2024-03-10T00:50:48.840 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:50:48 smithi106 ceph-mon[33460]: osdmap e42: 6 total, 6 up, 6 in 2024-03-10T00:50:48.840 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:50:48 smithi106 ceph-mon[33460]: from='client.24241 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi170:/dev/nvme2n1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-10T00:50:48.840 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:50:48 smithi106 ceph-mon[33460]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-10T00:50:48.840 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:50:48 smithi106 ceph-mon[33460]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-10T00:50:48.840 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:50:48 smithi106 ceph-mon[33460]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-10T00:50:50.774 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:50:50 smithi170 ceph-mon[27620]: pgmap v149: 1 pgs: 1 unknown; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-03-10T00:50:50.775 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:50:50 smithi170 ceph-mon[27620]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:50:50.775 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:50:50 smithi170 ceph-mon[27620]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:50:50.775 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:50:50 smithi170 ceph-mon[27620]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:50:50.775 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:50:50 smithi170 ceph-mon[27620]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-10T00:50:50.775 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:50:50 smithi170 ceph-mon[27620]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-10T00:50:50.775 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:50:50 smithi170 ceph-mon[27620]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-10T00:50:50.775 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:50:50 smithi170 ceph-mon[27620]: pgmap v150: 1 pgs: 1 unknown; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-03-10T00:50:50.775 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:50:50 smithi170 ceph-mon[27620]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:50:50.775 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:50:50 smithi170 ceph-mon[27620]: Health check cleared: CEPHADM_STRAY_DAEMON (was: 1 stray daemon(s) not managed by cephadm) 2024-03-10T00:50:50.775 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:50:50 smithi170 ceph-mon[27620]: Cluster is now healthy 2024-03-10T00:50:50.838 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:50:50 smithi106 ceph-mon[25742]: pgmap v149: 1 pgs: 1 unknown; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-03-10T00:50:50.839 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:50:50 smithi106 ceph-mon[25742]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:50:50.839 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:50:50 smithi106 ceph-mon[25742]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:50:50.839 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:50:50 smithi106 ceph-mon[25742]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:50:50.839 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:50:50 smithi106 ceph-mon[25742]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-10T00:50:50.839 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:50:50 smithi106 ceph-mon[25742]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-10T00:50:50.839 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:50:50 smithi106 ceph-mon[25742]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-10T00:50:50.839 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:50:50 smithi106 ceph-mon[25742]: pgmap v150: 1 pgs: 1 unknown; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-03-10T00:50:50.839 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:50:50 smithi106 ceph-mon[25742]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:50:50.839 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:50:50 smithi106 ceph-mon[25742]: Health check cleared: CEPHADM_STRAY_DAEMON (was: 1 stray daemon(s) not managed by cephadm) 2024-03-10T00:50:50.839 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:50:50 smithi106 ceph-mon[25742]: Cluster is now healthy 2024-03-10T00:50:50.839 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:50:50 smithi106 ceph-mon[33460]: pgmap v149: 1 pgs: 1 unknown; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-03-10T00:50:50.840 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:50:50 smithi106 ceph-mon[33460]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:50:50.840 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:50:50 smithi106 ceph-mon[33460]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:50:50.840 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:50:50 smithi106 ceph-mon[33460]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:50:50.840 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:50:50 smithi106 ceph-mon[33460]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-10T00:50:50.840 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:50:50 smithi106 ceph-mon[33460]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-10T00:50:50.840 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:50:50 smithi106 ceph-mon[33460]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-10T00:50:50.840 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:50:50 smithi106 ceph-mon[33460]: pgmap v150: 1 pgs: 1 unknown; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-03-10T00:50:50.840 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:50:50 smithi106 ceph-mon[33460]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:50:50.840 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:50:50 smithi106 ceph-mon[33460]: Health check cleared: CEPHADM_STRAY_DAEMON (was: 1 stray daemon(s) not managed by cephadm) 2024-03-10T00:50:50.840 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:50:50 smithi106 ceph-mon[33460]: Cluster is now healthy 2024-03-10T00:50:51.774 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:50:51 smithi170 ceph-mon[27620]: from='client.? 172.21.15.170:0/2560384222' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "46e00786-a5d2-48fe-b750-89beb0d63194"}]: dispatch 2024-03-10T00:50:51.774 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:50:51 smithi170 ceph-mon[27620]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "46e00786-a5d2-48fe-b750-89beb0d63194"}]: dispatch 2024-03-10T00:50:51.775 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:50:51 smithi170 ceph-mon[27620]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "46e00786-a5d2-48fe-b750-89beb0d63194"}]': finished 2024-03-10T00:50:51.775 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:50:51 smithi170 ceph-mon[27620]: osdmap e43: 7 total, 6 up, 7 in 2024-03-10T00:50:51.775 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:50:51 smithi170 ceph-mon[27620]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-10T00:50:51.838 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:50:51 smithi106 ceph-mon[25742]: from='client.? 172.21.15.170:0/2560384222' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "46e00786-a5d2-48fe-b750-89beb0d63194"}]: dispatch 2024-03-10T00:50:51.838 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:50:51 smithi106 ceph-mon[25742]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "46e00786-a5d2-48fe-b750-89beb0d63194"}]: dispatch 2024-03-10T00:50:51.839 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:50:51 smithi106 ceph-mon[25742]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "46e00786-a5d2-48fe-b750-89beb0d63194"}]': finished 2024-03-10T00:50:51.839 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:50:51 smithi106 ceph-mon[25742]: osdmap e43: 7 total, 6 up, 7 in 2024-03-10T00:50:51.839 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:50:51 smithi106 ceph-mon[25742]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-10T00:50:51.839 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:50:51 smithi106 ceph-mon[33460]: from='client.? 172.21.15.170:0/2560384222' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "46e00786-a5d2-48fe-b750-89beb0d63194"}]: dispatch 2024-03-10T00:50:51.839 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:50:51 smithi106 ceph-mon[33460]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "46e00786-a5d2-48fe-b750-89beb0d63194"}]: dispatch 2024-03-10T00:50:51.839 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:50:51 smithi106 ceph-mon[33460]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "46e00786-a5d2-48fe-b750-89beb0d63194"}]': finished 2024-03-10T00:50:51.839 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:50:51 smithi106 ceph-mon[33460]: osdmap e43: 7 total, 6 up, 7 in 2024-03-10T00:50:51.839 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:50:51 smithi106 ceph-mon[33460]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-10T00:50:52.775 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:50:52 smithi170 ceph-mon[27620]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 175 MiB used, 536 GiB / 536 GiB avail; 136 KiB/s, 0 objects/s recovering 2024-03-10T00:50:52.775 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:50:52 smithi170 ceph-mon[27620]: from='client.? 172.21.15.170:0/3391783251' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-10T00:50:52.838 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:50:52 smithi106 ceph-mon[25742]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 175 MiB used, 536 GiB / 536 GiB avail; 136 KiB/s, 0 objects/s recovering 2024-03-10T00:50:52.838 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:50:52 smithi106 ceph-mon[25742]: from='client.? 172.21.15.170:0/3391783251' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-10T00:50:52.839 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:50:52 smithi106 ceph-mon[33460]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 175 MiB used, 536 GiB / 536 GiB avail; 136 KiB/s, 0 objects/s recovering 2024-03-10T00:50:52.839 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:50:52 smithi106 ceph-mon[33460]: from='client.? 172.21.15.170:0/3391783251' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-10T00:50:54.687 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:50:54 smithi170 ceph-mon[27620]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail; 94 KiB/s, 0 objects/s recovering 2024-03-10T00:50:54.838 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:50:54 smithi106 ceph-mon[25742]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail; 94 KiB/s, 0 objects/s recovering 2024-03-10T00:50:54.839 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:50:54 smithi106 ceph-mon[33460]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail; 94 KiB/s, 0 objects/s recovering 2024-03-10T00:50:56.762 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:50:56 smithi170 ceph-mon[27620]: pgmap v154: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-03-10T00:50:56.838 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:50:56 smithi106 ceph-mon[25742]: pgmap v154: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-03-10T00:50:56.838 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:50:56 smithi106 ceph-mon[33460]: pgmap v154: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-03-10T00:50:58.624 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:50:58 smithi170 ceph-mon[27620]: pgmap v155: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail; 71 KiB/s, 0 objects/s recovering 2024-03-10T00:50:58.838 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:50:58 smithi106 ceph-mon[25742]: pgmap v155: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail; 71 KiB/s, 0 objects/s recovering 2024-03-10T00:50:58.839 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:50:58 smithi106 ceph-mon[33460]: pgmap v155: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail; 71 KiB/s, 0 objects/s recovering 2024-03-10T00:51:00.774 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:51:00 smithi106 ceph-mon[25742]: pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-03-10T00:51:00.775 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:51:00 smithi106 ceph-mon[25742]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:51:00.775 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:51:00 smithi106 ceph-mon[25742]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:51:00.775 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:51:00 smithi106 ceph-mon[25742]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:51:00.775 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:51:00 smithi106 ceph-mon[25742]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-03-10T00:51:00.775 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:51:00 smithi106 ceph-mon[25742]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-10T00:51:00.775 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:51:00 smithi106 ceph-mon[25742]: Deploying daemon osd.6 on smithi170 2024-03-10T00:51:00.775 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:51:00 smithi106 ceph-mon[33460]: pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-03-10T00:51:00.775 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:51:00 smithi106 ceph-mon[33460]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:51:00.775 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:51:00 smithi106 ceph-mon[33460]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:51:00.775 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:51:00 smithi106 ceph-mon[33460]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:51:00.775 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:51:00 smithi106 ceph-mon[33460]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-03-10T00:51:00.776 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:51:00 smithi106 ceph-mon[33460]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-10T00:51:00.776 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:51:00 smithi106 ceph-mon[33460]: Deploying daemon osd.6 on smithi170 2024-03-10T00:51:01.024 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:51:00 smithi170 ceph-mon[27620]: pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-03-10T00:51:01.024 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:51:00 smithi170 ceph-mon[27620]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:51:01.024 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:51:00 smithi170 ceph-mon[27620]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:51:01.025 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:51:00 smithi170 ceph-mon[27620]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:51:01.025 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:51:00 smithi170 ceph-mon[27620]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-03-10T00:51:01.025 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:51:00 smithi170 ceph-mon[27620]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-10T00:51:01.025 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:51:00 smithi170 ceph-mon[27620]: Deploying daemon osd.6 on smithi170 2024-03-10T00:51:02.774 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:51:02 smithi170 ceph-mon[27620]: pgmap v157: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-03-10T00:51:02.838 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:51:02 smithi106 ceph-mon[33460]: pgmap v157: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-03-10T00:51:02.839 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:51:02 smithi106 ceph-mon[25742]: pgmap v157: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-03-10T00:51:04.690 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:51:04 smithi170 ceph-mon[27620]: pgmap v158: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-03-10T00:51:04.838 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:51:04 smithi106 ceph-mon[25742]: pgmap v158: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-03-10T00:51:04.838 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:51:04 smithi106 ceph-mon[33460]: pgmap v158: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-03-10T00:51:06.588 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:51:06 smithi106 ceph-mon[25742]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:51:06.588 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:51:06 smithi106 ceph-mon[25742]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:51:06.589 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:51:06 smithi106 ceph-mon[25742]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-10T00:51:06.589 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:51:06 smithi106 ceph-mon[25742]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-10T00:51:06.589 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:51:06 smithi106 ceph-mon[25742]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-10T00:51:06.589 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:51:06 smithi106 ceph-mon[25742]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:51:06.589 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:51:06 smithi106 ceph-mon[25742]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-03-10T00:51:06.589 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:51:06 smithi106 ceph-mon[33460]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:51:06.589 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:51:06 smithi106 ceph-mon[33460]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:51:06.589 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:51:06 smithi106 ceph-mon[33460]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-10T00:51:06.589 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:51:06 smithi106 ceph-mon[33460]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-10T00:51:06.589 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:51:06 smithi106 ceph-mon[33460]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-10T00:51:06.589 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:51:06 smithi106 ceph-mon[33460]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:51:06.590 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:51:06 smithi106 ceph-mon[33460]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-03-10T00:51:06.674 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:51:06 smithi170 ceph-mon[27620]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:51:06.674 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:51:06 smithi170 ceph-mon[27620]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:51:06.674 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:51:06 smithi170 ceph-mon[27620]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-10T00:51:06.674 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:51:06 smithi170 ceph-mon[27620]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-10T00:51:06.675 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:51:06 smithi170 ceph-mon[27620]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-10T00:51:06.675 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:51:06 smithi170 ceph-mon[27620]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:51:06.675 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:51:06 smithi170 ceph-mon[27620]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-03-10T00:51:08.581 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:51:08 smithi170 ceph-mon[27620]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-03-10T00:51:08.581 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:51:08 smithi170 ceph-mon[27620]: from='osd.6 [v2:172.21.15.170:6816/814825309,v1:172.21.15.170:6817/814825309]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-03-10T00:51:08.582 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:51:08 smithi170 ceph-mon[27620]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-03-10T00:51:08.650 INFO:teuthology.orchestra.run.smithi170.stdout:Created osd(s) 6 on host 'smithi170' 2024-03-10T00:51:08.838 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:51:08 smithi106 ceph-mon[25742]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-03-10T00:51:08.838 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:51:08 smithi106 ceph-mon[25742]: from='osd.6 [v2:172.21.15.170:6816/814825309,v1:172.21.15.170:6817/814825309]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-03-10T00:51:08.838 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:51:08 smithi106 ceph-mon[25742]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-03-10T00:51:08.839 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:51:08 smithi106 ceph-mon[33460]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-03-10T00:51:08.839 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:51:08 smithi106 ceph-mon[33460]: from='osd.6 [v2:172.21.15.170:6816/814825309,v1:172.21.15.170:6817/814825309]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-03-10T00:51:08.839 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:51:08 smithi106 ceph-mon[33460]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-03-10T00:51:09.243 DEBUG:teuthology.orchestra.run.smithi170:osd.6> sudo journalctl -f -n 0 -u ceph-75c37498-de77-11ee-95c7-87774f69a715@osd.6.service 2024-03-10T00:51:09.246 INFO:tasks.cephadm:Deploying osd.7 on smithi170 with /dev/nvme1n1... 2024-03-10T00:51:09.246 DEBUG:teuthology.orchestra.run.smithi170:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:831bc188fa69f143bbba69ac76119e784500e8e3 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 75c37498-de77-11ee-95c7-87774f69a715 -- ceph-volume lvm zap /dev/nvme1n1 2024-03-10T00:51:09.565 INFO:teuthology.orchestra.run.smithi170.stderr:Inferring config /var/lib/ceph/75c37498-de77-11ee-95c7-87774f69a715/mon.b/config 2024-03-10T00:51:09.774 INFO:journalctl@ceph.osd.6.smithi170.stdout:Mar 10 00:51:09 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-osd-6[43049]: 2024-03-10T00:51:09.509+0000 7f5854e2d700 -1 osd.6 0 waiting for initial osdmap 2024-03-10T00:51:09.774 INFO:journalctl@ceph.osd.6.smithi170.stdout:Mar 10 00:51:09 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-osd-6[43049]: 2024-03-10T00:51:09.517+0000 7f5850c20700 -1 osd.6 45 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-10T00:51:09.775 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:51:09 smithi170 ceph-mon[27620]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-03-10T00:51:09.775 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:51:09 smithi170 ceph-mon[27620]: from='osd.6 [v2:172.21.15.170:6816/814825309,v1:172.21.15.170:6817/814825309]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi170", "root=default"]}]: dispatch 2024-03-10T00:51:09.775 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:51:09 smithi170 ceph-mon[27620]: osdmap e44: 7 total, 6 up, 7 in 2024-03-10T00:51:09.775 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:51:09 smithi170 ceph-mon[27620]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-10T00:51:09.775 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:51:09 smithi170 ceph-mon[27620]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi170", "root=default"]}]: dispatch 2024-03-10T00:51:09.775 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:51:09 smithi170 ceph-mon[27620]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:51:09.775 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:51:09 smithi170 ceph-mon[27620]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:51:09.775 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:51:09 smithi170 ceph-mon[27620]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-10T00:51:09.775 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:51:09 smithi170 ceph-mon[27620]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-10T00:51:09.775 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:51:09 smithi170 ceph-mon[27620]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-10T00:51:09.775 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:51:09 smithi170 ceph-mon[27620]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-10T00:51:09.775 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:51:09 smithi170 ceph-mon[27620]: Adjusting osd_memory_target on smithi170 to 5354M 2024-03-10T00:51:09.776 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:51:09 smithi170 ceph-mon[27620]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:51:09.776 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:51:09 smithi170 ceph-mon[27620]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-10T00:51:09.776 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:51:09 smithi170 ceph-mon[27620]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-10T00:51:09.776 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:51:09 smithi170 ceph-mon[27620]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:51:09.838 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:51:09 smithi106 ceph-mon[25742]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-03-10T00:51:09.839 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:51:09 smithi106 ceph-mon[25742]: from='osd.6 [v2:172.21.15.170:6816/814825309,v1:172.21.15.170:6817/814825309]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi170", "root=default"]}]: dispatch 2024-03-10T00:51:09.839 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:51:09 smithi106 ceph-mon[25742]: osdmap e44: 7 total, 6 up, 7 in 2024-03-10T00:51:09.840 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:51:09 smithi106 ceph-mon[25742]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-10T00:51:09.840 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:51:09 smithi106 ceph-mon[25742]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi170", "root=default"]}]: dispatch 2024-03-10T00:51:09.840 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:51:09 smithi106 ceph-mon[25742]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:51:09.840 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:51:09 smithi106 ceph-mon[25742]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:51:09.840 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:51:09 smithi106 ceph-mon[25742]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-10T00:51:09.840 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:51:09 smithi106 ceph-mon[25742]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-10T00:51:09.840 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:51:09 smithi106 ceph-mon[25742]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-10T00:51:09.840 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:51:09 smithi106 ceph-mon[25742]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-10T00:51:09.840 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:51:09 smithi106 ceph-mon[25742]: Adjusting osd_memory_target on smithi170 to 5354M 2024-03-10T00:51:09.840 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:51:09 smithi106 ceph-mon[25742]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:51:09.840 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:51:09 smithi106 ceph-mon[25742]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-10T00:51:09.840 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:51:09 smithi106 ceph-mon[25742]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-10T00:51:09.841 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:51:09 smithi106 ceph-mon[25742]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:51:09.841 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:51:09 smithi106 ceph-mon[33460]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-03-10T00:51:09.841 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:51:09 smithi106 ceph-mon[33460]: from='osd.6 [v2:172.21.15.170:6816/814825309,v1:172.21.15.170:6817/814825309]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi170", "root=default"]}]: dispatch 2024-03-10T00:51:09.841 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:51:09 smithi106 ceph-mon[33460]: osdmap e44: 7 total, 6 up, 7 in 2024-03-10T00:51:09.841 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:51:09 smithi106 ceph-mon[33460]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-10T00:51:09.841 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:51:09 smithi106 ceph-mon[33460]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi170", "root=default"]}]: dispatch 2024-03-10T00:51:09.841 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:51:09 smithi106 ceph-mon[33460]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:51:09.841 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:51:09 smithi106 ceph-mon[33460]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:51:09.842 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:51:09 smithi106 ceph-mon[33460]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-10T00:51:09.842 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:51:09 smithi106 ceph-mon[33460]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-10T00:51:09.842 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:51:09 smithi106 ceph-mon[33460]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-10T00:51:09.842 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:51:09 smithi106 ceph-mon[33460]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-10T00:51:09.842 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:51:09 smithi106 ceph-mon[33460]: Adjusting osd_memory_target on smithi170 to 5354M 2024-03-10T00:51:09.842 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:51:09 smithi106 ceph-mon[33460]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:51:09.842 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:51:09 smithi106 ceph-mon[33460]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-10T00:51:09.842 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:51:09 smithi106 ceph-mon[33460]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-10T00:51:09.842 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:51:09 smithi106 ceph-mon[33460]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:51:10.536 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:51:10 smithi170 ceph-mon[27620]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-03-10T00:51:10.536 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:51:10 smithi170 ceph-mon[27620]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi170", "root=default"]}]': finished 2024-03-10T00:51:10.536 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:51:10 smithi170 ceph-mon[27620]: osdmap e45: 7 total, 6 up, 7 in 2024-03-10T00:51:10.536 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:51:10 smithi170 ceph-mon[27620]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-10T00:51:10.536 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:51:10 smithi170 ceph-mon[27620]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-10T00:51:10.536 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:51:10 smithi170 ceph-mon[27620]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:51:10.536 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:51:10 smithi170 ceph-mon[27620]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:51:10.536 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:51:10 smithi170 ceph-mon[27620]: Detected new or changed devices on smithi170 2024-03-10T00:51:10.536 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:51:10 smithi170 ceph-mon[27620]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:51:10.536 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:51:10 smithi170 ceph-mon[27620]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-10T00:51:10.536 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:51:10 smithi170 ceph-mon[27620]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-10T00:51:10.537 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:51:10 smithi170 ceph-mon[27620]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-10T00:51:10.537 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:51:10 smithi170 ceph-mon[27620]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:51:10.537 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:51:10 smithi170 ceph-mon[27620]: osd.6 [v2:172.21.15.170:6816/814825309,v1:172.21.15.170:6817/814825309] boot 2024-03-10T00:51:10.537 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:51:10 smithi170 ceph-mon[27620]: osdmap e46: 7 total, 7 up, 7 in 2024-03-10T00:51:10.537 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:51:10 smithi170 ceph-mon[27620]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-10T00:51:10.838 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:51:10 smithi106 ceph-mon[25742]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-03-10T00:51:10.838 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:51:10 smithi106 ceph-mon[25742]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi170", "root=default"]}]': finished 2024-03-10T00:51:10.839 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:51:10 smithi106 ceph-mon[25742]: osdmap e45: 7 total, 6 up, 7 in 2024-03-10T00:51:10.839 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:51:10 smithi106 ceph-mon[25742]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-10T00:51:10.839 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:51:10 smithi106 ceph-mon[25742]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-10T00:51:10.839 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:51:10 smithi106 ceph-mon[25742]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:51:10.839 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:51:10 smithi106 ceph-mon[25742]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:51:10.839 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:51:10 smithi106 ceph-mon[25742]: Detected new or changed devices on smithi170 2024-03-10T00:51:10.839 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:51:10 smithi106 ceph-mon[25742]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:51:10.839 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:51:10 smithi106 ceph-mon[25742]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-10T00:51:10.839 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:51:10 smithi106 ceph-mon[25742]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-10T00:51:10.839 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:51:10 smithi106 ceph-mon[25742]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-10T00:51:10.839 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:51:10 smithi106 ceph-mon[25742]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:51:10.839 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:51:10 smithi106 ceph-mon[25742]: osd.6 [v2:172.21.15.170:6816/814825309,v1:172.21.15.170:6817/814825309] boot 2024-03-10T00:51:10.839 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:51:10 smithi106 ceph-mon[25742]: osdmap e46: 7 total, 7 up, 7 in 2024-03-10T00:51:10.839 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:51:10 smithi106 ceph-mon[25742]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-10T00:51:10.840 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:51:10 smithi106 ceph-mon[33460]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-03-10T00:51:10.840 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:51:10 smithi106 ceph-mon[33460]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi170", "root=default"]}]': finished 2024-03-10T00:51:10.840 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:51:10 smithi106 ceph-mon[33460]: osdmap e45: 7 total, 6 up, 7 in 2024-03-10T00:51:10.840 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:51:10 smithi106 ceph-mon[33460]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-10T00:51:10.840 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:51:10 smithi106 ceph-mon[33460]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-10T00:51:10.840 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:51:10 smithi106 ceph-mon[33460]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:51:10.840 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:51:10 smithi106 ceph-mon[33460]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:51:10.840 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:51:10 smithi106 ceph-mon[33460]: Detected new or changed devices on smithi170 2024-03-10T00:51:10.840 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:51:10 smithi106 ceph-mon[33460]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:51:10.840 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:51:10 smithi106 ceph-mon[33460]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-10T00:51:10.841 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:51:10 smithi106 ceph-mon[33460]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-10T00:51:10.841 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:51:10 smithi106 ceph-mon[33460]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-10T00:51:10.841 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:51:10 smithi106 ceph-mon[33460]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:51:10.841 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:51:10 smithi106 ceph-mon[33460]: osd.6 [v2:172.21.15.170:6816/814825309,v1:172.21.15.170:6817/814825309] boot 2024-03-10T00:51:10.841 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:51:10 smithi106 ceph-mon[33460]: osdmap e46: 7 total, 7 up, 7 in 2024-03-10T00:51:10.841 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:51:10 smithi106 ceph-mon[33460]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-10T00:51:11.792 INFO:teuthology.orchestra.run.smithi170.stderr:--> Zapping: /dev/nvme1n1 2024-03-10T00:51:11.792 INFO:teuthology.orchestra.run.smithi170.stderr:--> --destroy was not specified, but zapping a whole device will remove the partition table 2024-03-10T00:51:11.792 INFO:teuthology.orchestra.run.smithi170.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/nvme1n1 bs=1M count=10 conv=fsync 2024-03-10T00:51:11.792 INFO:teuthology.orchestra.run.smithi170.stderr: stderr: 10+0 records in 2024-03-10T00:51:11.792 INFO:teuthology.orchestra.run.smithi170.stderr:10+0 records out 2024-03-10T00:51:11.792 INFO:teuthology.orchestra.run.smithi170.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0108915 s, 963 MB/s 2024-03-10T00:51:11.793 INFO:teuthology.orchestra.run.smithi170.stderr:--> Zapping successful for: 2024-03-10T00:51:12.435 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:51:12 smithi170 ceph-mon[27620]: purged_snaps scrub starts 2024-03-10T00:51:12.435 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:51:12 smithi170 ceph-mon[27620]: purged_snaps scrub ok 2024-03-10T00:51:12.435 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:51:12 smithi170 ceph-mon[27620]: osdmap e47: 7 total, 7 up, 7 in 2024-03-10T00:51:12.435 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:51:12 smithi170 ceph-mon[27620]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 201 MiB used, 626 GiB / 626 GiB avail 2024-03-10T00:51:12.464 DEBUG:teuthology.orchestra.run.smithi170:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:831bc188fa69f143bbba69ac76119e784500e8e3 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 75c37498-de77-11ee-95c7-87774f69a715 -- ceph orch daemon add osd smithi170:/dev/nvme1n1 2024-03-10T00:51:12.588 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:51:12 smithi106 ceph-mon[25742]: purged_snaps scrub starts 2024-03-10T00:51:12.588 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:51:12 smithi106 ceph-mon[25742]: purged_snaps scrub ok 2024-03-10T00:51:12.588 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:51:12 smithi106 ceph-mon[25742]: osdmap e47: 7 total, 7 up, 7 in 2024-03-10T00:51:12.589 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:51:12 smithi106 ceph-mon[25742]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 201 MiB used, 626 GiB / 626 GiB avail 2024-03-10T00:51:12.589 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:51:12 smithi106 ceph-mon[33460]: purged_snaps scrub starts 2024-03-10T00:51:12.589 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:51:12 smithi106 ceph-mon[33460]: purged_snaps scrub ok 2024-03-10T00:51:12.589 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:51:12 smithi106 ceph-mon[33460]: osdmap e47: 7 total, 7 up, 7 in 2024-03-10T00:51:12.589 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:51:12 smithi106 ceph-mon[33460]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 201 MiB used, 626 GiB / 626 GiB avail 2024-03-10T00:51:12.722 INFO:teuthology.orchestra.run.smithi170.stderr:Inferring config /var/lib/ceph/75c37498-de77-11ee-95c7-87774f69a715/mon.b/config 2024-03-10T00:51:13.457 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:51:13 smithi170 ceph-mon[27620]: osdmap e48: 7 total, 7 up, 7 in 2024-03-10T00:51:13.457 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:51:13 smithi170 ceph-mon[27620]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:51:13.457 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:51:13 smithi170 ceph-mon[27620]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:51:13.457 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:51:13 smithi170 ceph-mon[27620]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:51:13.458 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:51:13 smithi170 ceph-mon[27620]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-10T00:51:13.458 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:51:13 smithi170 ceph-mon[27620]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-10T00:51:13.458 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:51:13 smithi170 ceph-mon[27620]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-10T00:51:13.458 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:51:13 smithi170 ceph-mon[27620]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:51:13.588 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:51:13 smithi106 ceph-mon[25742]: osdmap e48: 7 total, 7 up, 7 in 2024-03-10T00:51:13.588 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:51:13 smithi106 ceph-mon[25742]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:51:13.589 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:51:13 smithi106 ceph-mon[25742]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:51:13.589 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:51:13 smithi106 ceph-mon[25742]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:51:13.589 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:51:13 smithi106 ceph-mon[25742]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-10T00:51:13.589 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:51:13 smithi106 ceph-mon[25742]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-10T00:51:13.589 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:51:13 smithi106 ceph-mon[25742]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-10T00:51:13.589 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:51:13 smithi106 ceph-mon[25742]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:51:13.589 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:51:13 smithi106 ceph-mon[33460]: osdmap e48: 7 total, 7 up, 7 in 2024-03-10T00:51:13.589 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:51:13 smithi106 ceph-mon[33460]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:51:13.589 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:51:13 smithi106 ceph-mon[33460]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:51:13.589 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:51:13 smithi106 ceph-mon[33460]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:51:13.590 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:51:13 smithi106 ceph-mon[33460]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-10T00:51:13.590 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:51:13 smithi106 ceph-mon[33460]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-10T00:51:13.590 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:51:13 smithi106 ceph-mon[33460]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-10T00:51:13.590 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:51:13 smithi106 ceph-mon[33460]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:51:14.524 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:51:14 smithi170 ceph-mon[27620]: osdmap e49: 7 total, 7 up, 7 in 2024-03-10T00:51:14.525 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:51:14 smithi170 ceph-mon[27620]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 201 MiB used, 626 GiB / 626 GiB avail 2024-03-10T00:51:14.525 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:51:14 smithi170 ceph-mon[27620]: osdmap e50: 7 total, 7 up, 7 in 2024-03-10T00:51:14.588 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:51:14 smithi106 ceph-mon[25742]: osdmap e49: 7 total, 7 up, 7 in 2024-03-10T00:51:14.588 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:51:14 smithi106 ceph-mon[25742]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 201 MiB used, 626 GiB / 626 GiB avail 2024-03-10T00:51:14.588 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:51:14 smithi106 ceph-mon[25742]: osdmap e50: 7 total, 7 up, 7 in 2024-03-10T00:51:14.588 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:51:14 smithi106 ceph-mon[33460]: osdmap e49: 7 total, 7 up, 7 in 2024-03-10T00:51:14.589 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:51:14 smithi106 ceph-mon[33460]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 201 MiB used, 626 GiB / 626 GiB avail 2024-03-10T00:51:14.589 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:51:14 smithi106 ceph-mon[33460]: osdmap e50: 7 total, 7 up, 7 in 2024-03-10T00:51:15.678 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:51:15 smithi170 ceph-mon[27620]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-10T00:51:15.678 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:51:15 smithi170 ceph-mon[27620]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-10T00:51:15.678 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:51:15 smithi170 ceph-mon[27620]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-10T00:51:15.838 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:51:15 smithi106 ceph-mon[25742]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-10T00:51:15.839 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:51:15 smithi106 ceph-mon[25742]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-10T00:51:15.839 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:51:15 smithi106 ceph-mon[25742]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-10T00:51:15.839 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:51:15 smithi106 ceph-mon[33460]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-10T00:51:15.839 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:51:15 smithi106 ceph-mon[33460]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-10T00:51:15.839 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:51:15 smithi106 ceph-mon[33460]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-10T00:51:16.616 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:51:16 smithi170 ceph-mon[27620]: from='client.24268 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi170:/dev/nvme1n1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-10T00:51:16.616 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:51:16 smithi170 ceph-mon[27620]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 202 MiB used, 626 GiB / 626 GiB avail 2024-03-10T00:51:16.761 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:51:16 smithi106 ceph-mon[25742]: from='client.24268 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi170:/dev/nvme1n1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-10T00:51:16.761 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:51:16 smithi106 ceph-mon[25742]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 202 MiB used, 626 GiB / 626 GiB avail 2024-03-10T00:51:16.762 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:51:16 smithi106 ceph-mon[33460]: from='client.24268 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi170:/dev/nvme1n1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-10T00:51:16.762 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:51:16 smithi106 ceph-mon[33460]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 202 MiB used, 626 GiB / 626 GiB avail 2024-03-10T00:51:18.589 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:51:18 smithi106 ceph-mon[25742]: pgmap v172: 1 pgs: 1 active+clean; 577 KiB data, 202 MiB used, 626 GiB / 626 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-03-10T00:51:18.589 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:51:18 smithi106 ceph-mon[33460]: pgmap v172: 1 pgs: 1 active+clean; 577 KiB data, 202 MiB used, 626 GiB / 626 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-03-10T00:51:18.774 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:51:18 smithi170 ceph-mon[27620]: pgmap v172: 1 pgs: 1 active+clean; 577 KiB data, 202 MiB used, 626 GiB / 626 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-03-10T00:51:19.774 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:51:19 smithi170 ceph-mon[27620]: from='client.? 172.21.15.170:0/526435084' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "56567866-297b-457f-922a-7a53a1f3ef21"}]: dispatch 2024-03-10T00:51:19.775 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:51:19 smithi170 ceph-mon[27620]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "56567866-297b-457f-922a-7a53a1f3ef21"}]: dispatch 2024-03-10T00:51:19.775 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:51:19 smithi170 ceph-mon[27620]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "56567866-297b-457f-922a-7a53a1f3ef21"}]': finished 2024-03-10T00:51:19.775 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:51:19 smithi170 ceph-mon[27620]: osdmap e51: 8 total, 7 up, 8 in 2024-03-10T00:51:19.775 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:51:19 smithi170 ceph-mon[27620]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-10T00:51:19.837 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:51:19 smithi106 ceph-mon[33460]: from='client.? 172.21.15.170:0/526435084' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "56567866-297b-457f-922a-7a53a1f3ef21"}]: dispatch 2024-03-10T00:51:19.837 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:51:19 smithi106 ceph-mon[33460]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "56567866-297b-457f-922a-7a53a1f3ef21"}]: dispatch 2024-03-10T00:51:19.837 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:51:19 smithi106 ceph-mon[33460]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "56567866-297b-457f-922a-7a53a1f3ef21"}]': finished 2024-03-10T00:51:19.837 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:51:19 smithi106 ceph-mon[33460]: osdmap e51: 8 total, 7 up, 8 in 2024-03-10T00:51:19.837 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:51:19 smithi106 ceph-mon[33460]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-10T00:51:19.838 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:51:19 smithi106 ceph-mon[25742]: from='client.? 172.21.15.170:0/526435084' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "56567866-297b-457f-922a-7a53a1f3ef21"}]: dispatch 2024-03-10T00:51:19.838 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:51:19 smithi106 ceph-mon[25742]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "56567866-297b-457f-922a-7a53a1f3ef21"}]: dispatch 2024-03-10T00:51:19.838 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:51:19 smithi106 ceph-mon[25742]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "56567866-297b-457f-922a-7a53a1f3ef21"}]': finished 2024-03-10T00:51:19.838 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:51:19 smithi106 ceph-mon[25742]: osdmap e51: 8 total, 7 up, 8 in 2024-03-10T00:51:19.838 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:51:19 smithi106 ceph-mon[25742]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-10T00:51:20.838 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:51:20 smithi106 ceph-mon[25742]: pgmap v174: 1 pgs: 1 active+clean; 577 KiB data, 202 MiB used, 626 GiB / 626 GiB avail; 92 KiB/s, 0 objects/s recovering 2024-03-10T00:51:20.838 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:51:20 smithi106 ceph-mon[25742]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:51:20.839 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:51:20 smithi106 ceph-mon[25742]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:51:20.839 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:51:20 smithi106 ceph-mon[25742]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:51:20.839 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:51:20 smithi106 ceph-mon[25742]: from='client.? 172.21.15.170:0/4118031093' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-10T00:51:20.839 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:51:20 smithi106 ceph-mon[33460]: pgmap v174: 1 pgs: 1 active+clean; 577 KiB data, 202 MiB used, 626 GiB / 626 GiB avail; 92 KiB/s, 0 objects/s recovering 2024-03-10T00:51:20.839 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:51:20 smithi106 ceph-mon[33460]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:51:20.839 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:51:20 smithi106 ceph-mon[33460]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:51:20.839 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:51:20 smithi106 ceph-mon[33460]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:51:20.839 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:51:20 smithi106 ceph-mon[33460]: from='client.? 172.21.15.170:0/4118031093' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-10T00:51:21.024 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:51:20 smithi170 ceph-mon[27620]: pgmap v174: 1 pgs: 1 active+clean; 577 KiB data, 202 MiB used, 626 GiB / 626 GiB avail; 92 KiB/s, 0 objects/s recovering 2024-03-10T00:51:21.024 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:51:20 smithi170 ceph-mon[27620]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:51:21.024 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:51:20 smithi170 ceph-mon[27620]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:51:21.024 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:51:20 smithi170 ceph-mon[27620]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:51:21.025 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:51:20 smithi170 ceph-mon[27620]: from='client.? 172.21.15.170:0/4118031093' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-10T00:51:23.024 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:51:22 smithi170 ceph-mon[27620]: pgmap v175: 1 pgs: 1 active+clean; 577 KiB data, 202 MiB used, 626 GiB / 626 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-03-10T00:51:23.088 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:51:22 smithi106 ceph-mon[33460]: pgmap v175: 1 pgs: 1 active+clean; 577 KiB data, 202 MiB used, 626 GiB / 626 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-03-10T00:51:23.088 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:51:22 smithi106 ceph-mon[25742]: pgmap v175: 1 pgs: 1 active+clean; 577 KiB data, 202 MiB used, 626 GiB / 626 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-03-10T00:51:24.935 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:51:24 smithi170 ceph-mon[27620]: pgmap v176: 1 pgs: 1 active+clean; 577 KiB data, 202 MiB used, 626 GiB / 626 GiB avail; 62 KiB/s, 0 objects/s recovering 2024-03-10T00:51:25.088 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:51:24 smithi106 ceph-mon[33460]: pgmap v176: 1 pgs: 1 active+clean; 577 KiB data, 202 MiB used, 626 GiB / 626 GiB avail; 62 KiB/s, 0 objects/s recovering 2024-03-10T00:51:25.089 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:51:24 smithi106 ceph-mon[25742]: pgmap v176: 1 pgs: 1 active+clean; 577 KiB data, 202 MiB used, 626 GiB / 626 GiB avail; 62 KiB/s, 0 objects/s recovering 2024-03-10T00:51:26.940 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:51:26 smithi170 ceph-mon[27620]: pgmap v177: 1 pgs: 1 active+clean; 577 KiB data, 202 MiB used, 626 GiB / 626 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-03-10T00:51:26.941 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:51:26 smithi170 ceph-mon[27620]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-03-10T00:51:26.941 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:51:26 smithi170 ceph-mon[27620]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-10T00:51:27.088 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:51:26 smithi106 ceph-mon[33460]: pgmap v177: 1 pgs: 1 active+clean; 577 KiB data, 202 MiB used, 626 GiB / 626 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-03-10T00:51:27.088 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:51:26 smithi106 ceph-mon[33460]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-03-10T00:51:27.088 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:51:26 smithi106 ceph-mon[33460]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-10T00:51:27.089 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:51:26 smithi106 ceph-mon[25742]: pgmap v177: 1 pgs: 1 active+clean; 577 KiB data, 202 MiB used, 626 GiB / 626 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-03-10T00:51:27.089 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:51:26 smithi106 ceph-mon[25742]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-03-10T00:51:27.089 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:51:26 smithi106 ceph-mon[25742]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-10T00:51:28.024 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:51:27 smithi170 ceph-mon[27620]: Deploying daemon osd.7 on smithi170 2024-03-10T00:51:28.088 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:51:27 smithi106 ceph-mon[25742]: Deploying daemon osd.7 on smithi170 2024-03-10T00:51:28.088 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:51:27 smithi106 ceph-mon[33460]: Deploying daemon osd.7 on smithi170 2024-03-10T00:51:29.000 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:51:28 smithi170 ceph-mon[27620]: pgmap v178: 1 pgs: 1 active+clean; 577 KiB data, 202 MiB used, 626 GiB / 626 GiB avail 2024-03-10T00:51:29.088 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:51:28 smithi106 ceph-mon[25742]: pgmap v178: 1 pgs: 1 active+clean; 577 KiB data, 202 MiB used, 626 GiB / 626 GiB avail 2024-03-10T00:51:29.088 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:51:28 smithi106 ceph-mon[33460]: pgmap v178: 1 pgs: 1 active+clean; 577 KiB data, 202 MiB used, 626 GiB / 626 GiB avail 2024-03-10T00:51:30.968 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:51:30 smithi170 ceph-mon[27620]: pgmap v179: 1 pgs: 1 active+clean; 577 KiB data, 202 MiB used, 626 GiB / 626 GiB avail 2024-03-10T00:51:31.088 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:51:30 smithi106 ceph-mon[25742]: pgmap v179: 1 pgs: 1 active+clean; 577 KiB data, 202 MiB used, 626 GiB / 626 GiB avail 2024-03-10T00:51:31.088 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:51:30 smithi106 ceph-mon[33460]: pgmap v179: 1 pgs: 1 active+clean; 577 KiB data, 202 MiB used, 626 GiB / 626 GiB avail 2024-03-10T00:51:33.081 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:51:32 smithi170 ceph-mon[27620]: pgmap v180: 1 pgs: 1 active+clean; 577 KiB data, 202 MiB used, 626 GiB / 626 GiB avail 2024-03-10T00:51:33.082 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:51:32 smithi170 ceph-mon[27620]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:51:33.082 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:51:32 smithi170 ceph-mon[27620]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:51:33.082 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:51:32 smithi170 ceph-mon[27620]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-10T00:51:33.082 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:51:32 smithi170 ceph-mon[27620]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-10T00:51:33.082 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:51:32 smithi170 ceph-mon[27620]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-10T00:51:33.082 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:51:32 smithi170 ceph-mon[27620]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:51:33.088 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:51:32 smithi106 ceph-mon[25742]: pgmap v180: 1 pgs: 1 active+clean; 577 KiB data, 202 MiB used, 626 GiB / 626 GiB avail 2024-03-10T00:51:33.088 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:51:32 smithi106 ceph-mon[25742]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:51:33.088 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:51:32 smithi106 ceph-mon[25742]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:51:33.088 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:51:32 smithi106 ceph-mon[25742]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-10T00:51:33.088 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:51:32 smithi106 ceph-mon[25742]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-10T00:51:33.088 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:51:32 smithi106 ceph-mon[25742]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-10T00:51:33.088 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:51:32 smithi106 ceph-mon[25742]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:51:33.089 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:51:32 smithi106 ceph-mon[33460]: pgmap v180: 1 pgs: 1 active+clean; 577 KiB data, 202 MiB used, 626 GiB / 626 GiB avail 2024-03-10T00:51:33.089 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:51:32 smithi106 ceph-mon[33460]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:51:33.089 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:51:32 smithi106 ceph-mon[33460]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:51:33.089 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:51:32 smithi106 ceph-mon[33460]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-10T00:51:33.089 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:51:32 smithi106 ceph-mon[33460]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-10T00:51:33.089 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:51:32 smithi106 ceph-mon[33460]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-10T00:51:33.089 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:51:32 smithi106 ceph-mon[33460]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:51:34.024 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:51:33 smithi170 ceph-mon[27620]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:51:34.025 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:51:33 smithi170 ceph-mon[27620]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:51:34.025 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:51:33 smithi170 ceph-mon[27620]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:51:34.025 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:51:33 smithi170 ceph-mon[27620]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-10T00:51:34.025 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:51:33 smithi170 ceph-mon[27620]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-10T00:51:34.025 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:51:33 smithi170 ceph-mon[27620]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-10T00:51:34.025 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:51:33 smithi170 ceph-mon[27620]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:51:34.338 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:51:33 smithi106 ceph-mon[25742]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:51:34.338 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:51:33 smithi106 ceph-mon[25742]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:51:34.338 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:51:33 smithi106 ceph-mon[25742]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:51:34.339 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:51:33 smithi106 ceph-mon[25742]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-10T00:51:34.339 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:51:33 smithi106 ceph-mon[25742]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-10T00:51:34.339 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:51:33 smithi106 ceph-mon[25742]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-10T00:51:34.339 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:51:33 smithi106 ceph-mon[25742]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:51:34.339 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:51:33 smithi106 ceph-mon[33460]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:51:34.339 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:51:33 smithi106 ceph-mon[33460]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:51:34.339 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:51:33 smithi106 ceph-mon[33460]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:51:34.339 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:51:33 smithi106 ceph-mon[33460]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-10T00:51:34.339 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:51:33 smithi106 ceph-mon[33460]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-10T00:51:34.339 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:51:33 smithi106 ceph-mon[33460]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-10T00:51:34.339 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:51:33 smithi106 ceph-mon[33460]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:51:35.000 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:51:34 smithi170 ceph-mon[27620]: pgmap v181: 1 pgs: 1 active+clean; 577 KiB data, 202 MiB used, 626 GiB / 626 GiB avail 2024-03-10T00:51:35.000 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:51:34 smithi170 ceph-mon[27620]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:51:35.000 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:51:34 smithi170 ceph-mon[27620]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:51:35.000 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:51:34 smithi170 ceph-mon[27620]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:51:35.000 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:51:34 smithi170 ceph-mon[27620]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-10T00:51:35.000 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:51:34 smithi170 ceph-mon[27620]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-10T00:51:35.001 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:51:34 smithi170 ceph-mon[27620]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-10T00:51:35.001 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:51:34 smithi170 ceph-mon[27620]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:51:35.001 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:51:34 smithi170 ceph-mon[27620]: from='osd.7 [v2:172.21.15.170:6824/472398409,v1:172.21.15.170:6825/472398409]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-10T00:51:35.001 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:51:34 smithi170 ceph-mon[27620]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-10T00:51:35.338 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:51:34 smithi106 ceph-mon[25742]: pgmap v181: 1 pgs: 1 active+clean; 577 KiB data, 202 MiB used, 626 GiB / 626 GiB avail 2024-03-10T00:51:35.338 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:51:34 smithi106 ceph-mon[25742]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:51:35.339 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:51:34 smithi106 ceph-mon[25742]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:51:35.339 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:51:34 smithi106 ceph-mon[25742]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:51:35.339 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:51:34 smithi106 ceph-mon[25742]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-10T00:51:35.339 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:51:34 smithi106 ceph-mon[25742]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-10T00:51:35.339 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:51:34 smithi106 ceph-mon[25742]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-10T00:51:35.339 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:51:34 smithi106 ceph-mon[25742]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:51:35.339 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:51:34 smithi106 ceph-mon[25742]: from='osd.7 [v2:172.21.15.170:6824/472398409,v1:172.21.15.170:6825/472398409]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-10T00:51:35.339 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:51:34 smithi106 ceph-mon[25742]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-10T00:51:35.339 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:51:34 smithi106 ceph-mon[33460]: pgmap v181: 1 pgs: 1 active+clean; 577 KiB data, 202 MiB used, 626 GiB / 626 GiB avail 2024-03-10T00:51:35.339 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:51:34 smithi106 ceph-mon[33460]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:51:35.339 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:51:34 smithi106 ceph-mon[33460]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:51:35.339 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:51:34 smithi106 ceph-mon[33460]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:51:35.340 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:51:34 smithi106 ceph-mon[33460]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-10T00:51:35.340 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:51:34 smithi106 ceph-mon[33460]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-10T00:51:35.340 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:51:34 smithi106 ceph-mon[33460]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-10T00:51:35.340 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:51:34 smithi106 ceph-mon[33460]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:51:35.340 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:51:34 smithi106 ceph-mon[33460]: from='osd.7 [v2:172.21.15.170:6824/472398409,v1:172.21.15.170:6825/472398409]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-10T00:51:35.340 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:51:34 smithi106 ceph-mon[33460]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-10T00:51:35.991 INFO:teuthology.orchestra.run.smithi170.stdout:Created osd(s) 7 on host 'smithi170' 2024-03-10T00:51:36.837 DEBUG:teuthology.orchestra.run.smithi170:osd.7> sudo journalctl -f -n 0 -u ceph-75c37498-de77-11ee-95c7-87774f69a715@osd.7.service 2024-03-10T00:51:36.840 INFO:tasks.cephadm:Waiting for 8 OSDs to come up... 2024-03-10T00:51:36.840 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:831bc188fa69f143bbba69ac76119e784500e8e3 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 75c37498-de77-11ee-95c7-87774f69a715 -- ceph osd stat -f json 2024-03-10T00:51:37.078 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:51:36 smithi170 ceph-mon[27620]: pgmap v182: 1 pgs: 1 active+clean; 577 KiB data, 202 MiB used, 626 GiB / 626 GiB avail 2024-03-10T00:51:37.078 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:51:36 smithi170 ceph-mon[27620]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-03-10T00:51:37.078 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:51:36 smithi170 ceph-mon[27620]: from='osd.7 [v2:172.21.15.170:6824/472398409,v1:172.21.15.170:6825/472398409]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi170", "root=default"]}]: dispatch 2024-03-10T00:51:37.078 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:51:36 smithi170 ceph-mon[27620]: osdmap e52: 8 total, 7 up, 8 in 2024-03-10T00:51:37.078 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:51:36 smithi170 ceph-mon[27620]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-10T00:51:37.078 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:51:36 smithi170 ceph-mon[27620]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi170", "root=default"]}]: dispatch 2024-03-10T00:51:37.078 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:51:36 smithi170 ceph-mon[27620]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:51:37.078 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:51:36 smithi170 ceph-mon[27620]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-10T00:51:37.078 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:51:36 smithi170 ceph-mon[27620]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:51:37.078 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:51:36 smithi170 ceph-mon[27620]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-10T00:51:37.079 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:51:36 smithi170 ceph-mon[27620]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-10T00:51:37.079 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:51:36 smithi170 ceph-mon[27620]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-10T00:51:37.079 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:51:36 smithi170 ceph-mon[27620]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-10T00:51:37.079 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:51:36 smithi170 ceph-mon[27620]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:51:37.079 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:51:36 smithi170 ceph-mon[27620]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-10T00:51:37.079 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:51:36 smithi170 ceph-mon[27620]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-10T00:51:37.079 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:51:36 smithi170 ceph-mon[27620]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:51:37.090 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:51:36 smithi106 ceph-mon[33460]: pgmap v182: 1 pgs: 1 active+clean; 577 KiB data, 202 MiB used, 626 GiB / 626 GiB avail 2024-03-10T00:51:37.090 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:51:36 smithi106 ceph-mon[33460]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-03-10T00:51:37.090 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:51:36 smithi106 ceph-mon[33460]: from='osd.7 [v2:172.21.15.170:6824/472398409,v1:172.21.15.170:6825/472398409]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi170", "root=default"]}]: dispatch 2024-03-10T00:51:37.090 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:51:36 smithi106 ceph-mon[33460]: osdmap e52: 8 total, 7 up, 8 in 2024-03-10T00:51:37.090 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:51:36 smithi106 ceph-mon[33460]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-10T00:51:37.090 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:51:36 smithi106 ceph-mon[33460]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi170", "root=default"]}]: dispatch 2024-03-10T00:51:37.090 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:51:36 smithi106 ceph-mon[33460]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:51:37.091 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:51:36 smithi106 ceph-mon[33460]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-10T00:51:37.091 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:51:36 smithi106 ceph-mon[33460]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:51:37.091 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:51:36 smithi106 ceph-mon[33460]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-10T00:51:37.091 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:51:36 smithi106 ceph-mon[33460]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-10T00:51:37.091 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:51:36 smithi106 ceph-mon[33460]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-10T00:51:37.091 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:51:36 smithi106 ceph-mon[33460]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-10T00:51:37.091 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:51:36 smithi106 ceph-mon[33460]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:51:37.091 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:51:36 smithi106 ceph-mon[33460]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-10T00:51:37.091 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:51:36 smithi106 ceph-mon[33460]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-10T00:51:37.091 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:51:36 smithi106 ceph-mon[33460]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:51:37.092 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:51:36 smithi106 ceph-mon[25742]: pgmap v182: 1 pgs: 1 active+clean; 577 KiB data, 202 MiB used, 626 GiB / 626 GiB avail 2024-03-10T00:51:37.092 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:51:36 smithi106 ceph-mon[25742]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-03-10T00:51:37.092 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:51:36 smithi106 ceph-mon[25742]: from='osd.7 [v2:172.21.15.170:6824/472398409,v1:172.21.15.170:6825/472398409]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi170", "root=default"]}]: dispatch 2024-03-10T00:51:37.092 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:51:36 smithi106 ceph-mon[25742]: osdmap e52: 8 total, 7 up, 8 in 2024-03-10T00:51:37.092 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:51:36 smithi106 ceph-mon[25742]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-10T00:51:37.092 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:51:36 smithi106 ceph-mon[25742]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi170", "root=default"]}]: dispatch 2024-03-10T00:51:37.092 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:51:36 smithi106 ceph-mon[25742]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:51:37.092 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:51:36 smithi106 ceph-mon[25742]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-10T00:51:37.092 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:51:36 smithi106 ceph-mon[25742]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:51:37.092 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:51:36 smithi106 ceph-mon[25742]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-10T00:51:37.092 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:51:36 smithi106 ceph-mon[25742]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-10T00:51:37.093 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:51:36 smithi106 ceph-mon[25742]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-10T00:51:37.093 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:51:36 smithi106 ceph-mon[25742]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-10T00:51:37.093 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:51:36 smithi106 ceph-mon[25742]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:51:37.093 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:51:36 smithi106 ceph-mon[25742]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-10T00:51:37.093 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:51:36 smithi106 ceph-mon[25742]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-10T00:51:37.093 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:51:36 smithi106 ceph-mon[25742]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:51:37.093 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/75c37498-de77-11ee-95c7-87774f69a715/mon.a/config 2024-03-10T00:51:37.524 INFO:journalctl@ceph.osd.7.smithi170.stdout:Mar 10 00:51:37 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-osd-7[47108]: 2024-03-10T00:51:37.079+0000 7f7b8818b700 -1 osd.7 0 waiting for initial osdmap 2024-03-10T00:51:37.525 INFO:journalctl@ceph.osd.7.smithi170.stdout:Mar 10 00:51:37 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-osd-7[47108]: 2024-03-10T00:51:37.086+0000 7f7b7ff76700 -1 osd.7 53 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-10T00:51:37.838 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:51:37 smithi106 ceph-mon[25742]: Adjusting osd_memory_target on smithi170 to 4016M 2024-03-10T00:51:37.838 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:51:37 smithi106 ceph-mon[25742]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi170", "root=default"]}]': finished 2024-03-10T00:51:37.839 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:51:37 smithi106 ceph-mon[25742]: osdmap e53: 8 total, 7 up, 8 in 2024-03-10T00:51:37.839 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:51:37 smithi106 ceph-mon[25742]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-10T00:51:37.839 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:51:37 smithi106 ceph-mon[25742]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-10T00:51:37.839 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:51:37 smithi106 ceph-mon[25742]: from='osd.7 ' entity='osd.7' 2024-03-10T00:51:37.839 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:51:37 smithi106 ceph-mon[25742]: osd.7 [v2:172.21.15.170:6824/472398409,v1:172.21.15.170:6825/472398409] boot 2024-03-10T00:51:37.839 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:51:37 smithi106 ceph-mon[25742]: osdmap e54: 8 total, 8 up, 8 in 2024-03-10T00:51:37.839 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:51:37 smithi106 ceph-mon[25742]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-10T00:51:37.839 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:51:37 smithi106 ceph-mon[33460]: Adjusting osd_memory_target on smithi170 to 4016M 2024-03-10T00:51:37.839 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:51:37 smithi106 ceph-mon[33460]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi170", "root=default"]}]': finished 2024-03-10T00:51:37.839 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:51:37 smithi106 ceph-mon[33460]: osdmap e53: 8 total, 7 up, 8 in 2024-03-10T00:51:37.839 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:51:37 smithi106 ceph-mon[33460]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-10T00:51:37.839 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:51:37 smithi106 ceph-mon[33460]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-10T00:51:37.839 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:51:37 smithi106 ceph-mon[33460]: from='osd.7 ' entity='osd.7' 2024-03-10T00:51:37.840 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:51:37 smithi106 ceph-mon[33460]: osd.7 [v2:172.21.15.170:6824/472398409,v1:172.21.15.170:6825/472398409] boot 2024-03-10T00:51:37.840 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:51:37 smithi106 ceph-mon[33460]: osdmap e54: 8 total, 8 up, 8 in 2024-03-10T00:51:37.840 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:51:37 smithi106 ceph-mon[33460]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-10T00:51:37.998 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:51:37 smithi170 ceph-mon[27620]: Adjusting osd_memory_target on smithi170 to 4016M 2024-03-10T00:51:37.998 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:51:37 smithi170 ceph-mon[27620]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi170", "root=default"]}]': finished 2024-03-10T00:51:37.998 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:51:37 smithi170 ceph-mon[27620]: osdmap e53: 8 total, 7 up, 8 in 2024-03-10T00:51:37.998 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:51:37 smithi170 ceph-mon[27620]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-10T00:51:37.998 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:51:37 smithi170 ceph-mon[27620]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-10T00:51:37.998 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:51:37 smithi170 ceph-mon[27620]: from='osd.7 ' entity='osd.7' 2024-03-10T00:51:37.998 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:51:37 smithi170 ceph-mon[27620]: osd.7 [v2:172.21.15.170:6824/472398409,v1:172.21.15.170:6825/472398409] boot 2024-03-10T00:51:37.999 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:51:37 smithi170 ceph-mon[27620]: osdmap e54: 8 total, 8 up, 8 in 2024-03-10T00:51:37.999 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:51:37 smithi170 ceph-mon[27620]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-10T00:51:38.887 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:51:38 smithi106 ceph-mon[33460]: purged_snaps scrub starts 2024-03-10T00:51:38.887 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:51:38 smithi106 ceph-mon[33460]: purged_snaps scrub ok 2024-03-10T00:51:38.887 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:51:38 smithi106 ceph-mon[33460]: pgmap v185: 1 pgs: 1 active+clean; 577 KiB data, 202 MiB used, 626 GiB / 626 GiB avail 2024-03-10T00:51:38.887 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:51:38 smithi106 ceph-mon[33460]: osdmap e55: 8 total, 8 up, 8 in 2024-03-10T00:51:38.887 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:51:38 smithi106 ceph-mon[25742]: purged_snaps scrub starts 2024-03-10T00:51:38.887 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:51:38 smithi106 ceph-mon[25742]: purged_snaps scrub ok 2024-03-10T00:51:38.888 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:51:38 smithi106 ceph-mon[25742]: pgmap v185: 1 pgs: 1 active+clean; 577 KiB data, 202 MiB used, 626 GiB / 626 GiB avail 2024-03-10T00:51:38.888 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:51:38 smithi106 ceph-mon[25742]: osdmap e55: 8 total, 8 up, 8 in 2024-03-10T00:51:39.102 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-03-10T00:51:39.274 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:51:38 smithi170 ceph-mon[27620]: purged_snaps scrub starts 2024-03-10T00:51:39.274 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:51:38 smithi170 ceph-mon[27620]: purged_snaps scrub ok 2024-03-10T00:51:39.274 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:51:38 smithi170 ceph-mon[27620]: pgmap v185: 1 pgs: 1 active+clean; 577 KiB data, 202 MiB used, 626 GiB / 626 GiB avail 2024-03-10T00:51:39.274 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:51:38 smithi170 ceph-mon[27620]: osdmap e55: 8 total, 8 up, 8 in 2024-03-10T00:51:39.692 INFO:teuthology.orchestra.run.smithi106.stdout:{"epoch":55,"num_osds":8,"num_up_osds":8,"osd_up_since":1710031897,"num_in_osds":8,"osd_in_since":1710031878,"num_remapped_pgs":1} 2024-03-10T00:51:39.693 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:831bc188fa69f143bbba69ac76119e784500e8e3 shell --fsid 75c37498-de77-11ee-95c7-87774f69a715 -- ceph osd dump --format=json 2024-03-10T00:51:39.952 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/75c37498-de77-11ee-95c7-87774f69a715/mon.a/config 2024-03-10T00:51:40.073 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:51:39 smithi106 ceph-mon[25742]: from='client.? 172.21.15.106:0/2899976350' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-03-10T00:51:40.074 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:51:39 smithi106 ceph-mon[33460]: from='client.? 172.21.15.106:0/2899976350' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-03-10T00:51:40.176 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:51:39 smithi170 ceph-mon[27620]: from='client.? 172.21.15.106:0/2899976350' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-03-10T00:51:41.024 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:51:40 smithi170 ceph-mon[27620]: pgmap v188: 1 pgs: 1 remapped+peering; 577 KiB data, 356 MiB used, 715 GiB / 715 GiB avail 2024-03-10T00:51:41.025 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:51:40 smithi170 ceph-mon[27620]: osdmap e56: 8 total, 8 up, 8 in 2024-03-10T00:51:41.025 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:51:40 smithi170 ceph-mon[27620]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:51:41.025 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:51:40 smithi170 ceph-mon[27620]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:51:41.025 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:51:40 smithi170 ceph-mon[27620]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:51:41.025 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:51:40 smithi170 ceph-mon[27620]: osdmap e57: 8 total, 8 up, 8 in 2024-03-10T00:51:41.088 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:51:40 smithi106 ceph-mon[33460]: pgmap v188: 1 pgs: 1 remapped+peering; 577 KiB data, 356 MiB used, 715 GiB / 715 GiB avail 2024-03-10T00:51:41.089 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:51:40 smithi106 ceph-mon[33460]: osdmap e56: 8 total, 8 up, 8 in 2024-03-10T00:51:41.089 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:51:40 smithi106 ceph-mon[33460]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:51:41.089 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:51:40 smithi106 ceph-mon[33460]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:51:41.089 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:51:40 smithi106 ceph-mon[33460]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:51:41.089 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:51:40 smithi106 ceph-mon[33460]: osdmap e57: 8 total, 8 up, 8 in 2024-03-10T00:51:41.089 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:51:40 smithi106 ceph-mon[25742]: pgmap v188: 1 pgs: 1 remapped+peering; 577 KiB data, 356 MiB used, 715 GiB / 715 GiB avail 2024-03-10T00:51:41.089 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:51:40 smithi106 ceph-mon[25742]: osdmap e56: 8 total, 8 up, 8 in 2024-03-10T00:51:41.089 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:51:40 smithi106 ceph-mon[25742]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:51:41.089 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:51:40 smithi106 ceph-mon[25742]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:51:41.089 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:51:40 smithi106 ceph-mon[25742]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:51:41.090 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:51:40 smithi106 ceph-mon[25742]: osdmap e57: 8 total, 8 up, 8 in 2024-03-10T00:51:42.392 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-03-10T00:51:42.392 INFO:teuthology.orchestra.run.smithi106.stdout:{"epoch":58,"fsid":"75c37498-de77-11ee-95c7-87774f69a715","created":"2024-03-10T00:46:03.418276+0000","modified":"2024-03-10T00:51:41.180740+0000","last_up_change":"2024-03-10T00:51:37.801567+0000","last_in_change":"2024-03-10T00:51:18.974575+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":18,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-10T00:49:25.071132+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":7.8899998664855957,"score_stable":7.8899998664855957,"optimal_score":0.37999999523162842,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"90836761-e042-415d-b610-3a8383a65d57","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6802","nonce":3397660587},{"type":"v1","addr":"172.21.15.106:6803","nonce":3397660587}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6804","nonce":3397660587},{"type":"v1","addr":"172.21.15.106:6805","nonce":3397660587}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6808","nonce":3397660587},{"type":"v1","addr":"172.21.15.106:6809","nonce":3397660587}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6806","nonce":3397660587},{"type":"v1","addr":"172.21.15.106:6807","nonce":3397660587}]},"public_addr":"172.21.15.106:6803/3397660587","cluster_addr":"172.21.15.106:6805/3397660587","heartbeat_back_addr":"172.21.15.106:6809/3397660587","heartbeat_front_addr":"172.21.15.106:6807/3397660587","state":["exists","up"]},{"osd":1,"uuid":"207e1815-38e1-498c-b0cd-602a70cdffa1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6810","nonce":270281423},{"type":"v1","addr":"172.21.15.106:6811","nonce":270281423}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6812","nonce":270281423},{"type":"v1","addr":"172.21.15.106:6813","nonce":270281423}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6816","nonce":270281423},{"type":"v1","addr":"172.21.15.106:6817","nonce":270281423}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6814","nonce":270281423},{"type":"v1","addr":"172.21.15.106:6815","nonce":270281423}]},"public_addr":"172.21.15.106:6811/270281423","cluster_addr":"172.21.15.106:6813/270281423","heartbeat_back_addr":"172.21.15.106:6817/270281423","heartbeat_front_addr":"172.21.15.106:6815/270281423","state":["exists","up"]},{"osd":2,"uuid":"b8113c87-8517-4b71-b6b4-ad7f2fce167b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6818","nonce":3393056025},{"type":"v1","addr":"172.21.15.106:6819","nonce":3393056025}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6820","nonce":3393056025},{"type":"v1","addr":"172.21.15.106:6821","nonce":3393056025}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6824","nonce":3393056025},{"type":"v1","addr":"172.21.15.106:6825","nonce":3393056025}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6822","nonce":3393056025},{"type":"v1","addr":"172.21.15.106:6823","nonce":3393056025}]},"public_addr":"172.21.15.106:6819/3393056025","cluster_addr":"172.21.15.106:6821/3393056025","heartbeat_back_addr":"172.21.15.106:6825/3393056025","heartbeat_front_addr":"172.21.15.106:6823/3393056025","state":["exists","up"]},{"osd":3,"uuid":"b548c1c5-2658-47f6-89a3-f52f3e8c0758","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6826","nonce":2289173680},{"type":"v1","addr":"172.21.15.106:6827","nonce":2289173680}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6828","nonce":2289173680},{"type":"v1","addr":"172.21.15.106:6829","nonce":2289173680}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6832","nonce":2289173680},{"type":"v1","addr":"172.21.15.106:6833","nonce":2289173680}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6830","nonce":2289173680},{"type":"v1","addr":"172.21.15.106:6831","nonce":2289173680}]},"public_addr":"172.21.15.106:6827/2289173680","cluster_addr":"172.21.15.106:6829/2289173680","heartbeat_back_addr":"172.21.15.106:6833/2289173680","heartbeat_front_addr":"172.21.15.106:6831/2289173680","state":["exists","up"]},{"osd":4,"uuid":"4fcacd32-e9ca-4aa7-b99e-bac33fc58754","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":30,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6800","nonce":335809953},{"type":"v1","addr":"172.21.15.170:6801","nonce":335809953}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6802","nonce":335809953},{"type":"v1","addr":"172.21.15.170:6803","nonce":335809953}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6806","nonce":335809953},{"type":"v1","addr":"172.21.15.170:6807","nonce":335809953}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6804","nonce":335809953},{"type":"v1","addr":"172.21.15.170:6805","nonce":335809953}]},"public_addr":"172.21.15.170:6801/335809953","cluster_addr":"172.21.15.170:6803/335809953","heartbeat_back_addr":"172.21.15.170:6807/335809953","heartbeat_front_addr":"172.21.15.170:6805/335809953","state":["exists","up"]},{"osd":5,"uuid":"cf14b75a-d753-4279-9b67-593fed3c5785","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6808","nonce":47584024},{"type":"v1","addr":"172.21.15.170:6809","nonce":47584024}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6810","nonce":47584024},{"type":"v1","addr":"172.21.15.170:6811","nonce":47584024}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6814","nonce":47584024},{"type":"v1","addr":"172.21.15.170:6815","nonce":47584024}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6812","nonce":47584024},{"type":"v1","addr":"172.21.15.170:6813","nonce":47584024}]},"public_addr":"172.21.15.170:6809/47584024","cluster_addr":"172.21.15.170:6811/47584024","heartbeat_back_addr":"172.21.15.170:6815/47584024","heartbeat_front_addr":"172.21.15.170:6813/47584024","state":["exists","up"]},{"osd":6,"uuid":"46e00786-a5d2-48fe-b750-89beb0d63194","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":49,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6816","nonce":814825309},{"type":"v1","addr":"172.21.15.170:6817","nonce":814825309}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6818","nonce":814825309},{"type":"v1","addr":"172.21.15.170:6819","nonce":814825309}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6822","nonce":814825309},{"type":"v1","addr":"172.21.15.170:6823","nonce":814825309}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6820","nonce":814825309},{"type":"v1","addr":"172.21.15.170:6821","nonce":814825309}]},"public_addr":"172.21.15.170:6817/814825309","cluster_addr":"172.21.15.170:6819/814825309","heartbeat_back_addr":"172.21.15.170:6823/814825309","heartbeat_front_addr":"172.21.15.170:6821/814825309","state":["exists","up"]},{"osd":7,"uuid":"56567866-297b-457f-922a-7a53a1f3ef21","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":54,"up_thru":57,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6824","nonce":472398409},{"type":"v1","addr":"172.21.15.170:6825","nonce":472398409}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6826","nonce":472398409},{"type":"v1","addr":"172.21.15.170:6827","nonce":472398409}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6830","nonce":472398409},{"type":"v1","addr":"172.21.15.170:6831","nonce":472398409}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6828","nonce":472398409},{"type":"v1","addr":"172.21.15.170:6829","nonce":472398409}]},"public_addr":"172.21.15.170:6825/472398409","cluster_addr":"172.21.15.170:6827/472398409","heartbeat_back_addr":"172.21.15.170:6831/472398409","heartbeat_front_addr":"172.21.15.170:6829/472398409","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-10T00:48:30.232907+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-10T00:48:56.221202+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-10T00:49:22.529778+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-10T00:49:52.090754+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-10T00:50:16.772217+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-10T00:50:42.035375+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-10T00:51:09.412164+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-10T00:51:35.931598+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.106:0/3982476751":"2024-03-11T00:47:00.732937+0000","172.21.15.106:6801/2449347734":"2024-03-11T00:47:00.732937+0000","172.21.15.106:6800/2449347734":"2024-03-11T00:47:00.732937+0000","172.21.15.106:0/3555678354":"2024-03-11T00:47:00.732937+0000","172.21.15.106:6801/1073099015":"2024-03-11T00:46:33.968987+0000","172.21.15.106:0/4150429375":"2024-03-11T00:46:33.968987+0000","172.21.15.106:6800/1073099015":"2024-03-11T00:46:33.968987+0000","172.21.15.106:0/232887326":"2024-03-11T00:46:33.968987+0000","172.21.15.106:0/4228752537":"2024-03-11T00:46:33.968987+0000","172.21.15.106:0/3721367154":"2024-03-11T00:47:00.732937+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-10T00:51:42.524 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:51:42 smithi170 ceph-mon[27620]: osdmap e58: 8 total, 8 up, 8 in 2024-03-10T00:51:42.524 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:51:42 smithi170 ceph-mon[27620]: pgmap v192: 1 pgs: 1 remapped+peering; 577 KiB data, 357 MiB used, 715 GiB / 715 GiB avail 2024-03-10T00:51:42.563 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:51:42 smithi106 ceph-mon[25742]: osdmap e58: 8 total, 8 up, 8 in 2024-03-10T00:51:42.563 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:51:42 smithi106 ceph-mon[25742]: pgmap v192: 1 pgs: 1 remapped+peering; 577 KiB data, 357 MiB used, 715 GiB / 715 GiB avail 2024-03-10T00:51:42.564 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:51:42 smithi106 ceph-mon[33460]: osdmap e58: 8 total, 8 up, 8 in 2024-03-10T00:51:42.564 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:51:42 smithi106 ceph-mon[33460]: pgmap v192: 1 pgs: 1 remapped+peering; 577 KiB data, 357 MiB used, 715 GiB / 715 GiB avail 2024-03-10T00:51:43.496 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:51:43 smithi106 ceph-mon[25742]: from='client.? 172.21.15.106:0/3972551761' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-10T00:51:43.497 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:51:43 smithi106 ceph-mon[33460]: from='client.? 172.21.15.106:0/3972551761' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-10T00:51:43.524 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:51:43 smithi170 ceph-mon[27620]: from='client.? 172.21.15.106:0/3972551761' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-10T00:51:43.790 INFO:tasks.cephadm.ceph_manager.ceph:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-10T00:49:25.071132+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 7.889999866485596, 'score_stable': 7.889999866485596, 'optimal_score': 0.3799999952316284, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-10T00:51:43.791 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:831bc188fa69f143bbba69ac76119e784500e8e3 shell --fsid 75c37498-de77-11ee-95c7-87774f69a715 -- ceph osd pool get .mgr pg_num 2024-03-10T00:51:44.490 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:51:44 smithi106 ceph-mon[25742]: pgmap v193: 1 pgs: 1 active+clean; 577 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 124 KiB/s, 0 objects/s recovering 2024-03-10T00:51:44.490 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:51:44 smithi106 ceph-mon[33460]: pgmap v193: 1 pgs: 1 active+clean; 577 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 124 KiB/s, 0 objects/s recovering 2024-03-10T00:51:44.524 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:51:44 smithi170 ceph-mon[27620]: pgmap v193: 1 pgs: 1 active+clean; 577 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 124 KiB/s, 0 objects/s recovering 2024-03-10T00:51:44.525 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/75c37498-de77-11ee-95c7-87774f69a715/mon.a/config 2024-03-10T00:51:46.283 INFO:teuthology.orchestra.run.smithi106.stdout:pg_num: 1 2024-03-10T00:51:46.588 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:51:46 smithi106 ceph-mon[25742]: pgmap v194: 1 pgs: 1 active+clean; 577 KiB data, 229 MiB used, 715 GiB / 715 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-03-10T00:51:46.588 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:51:46 smithi106 ceph-mon[25742]: from='client.? 172.21.15.106:0/2098723210' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-10T00:51:46.588 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:51:46 smithi106 ceph-mon[33460]: pgmap v194: 1 pgs: 1 active+clean; 577 KiB data, 229 MiB used, 715 GiB / 715 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-03-10T00:51:46.589 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:51:46 smithi106 ceph-mon[33460]: from='client.? 172.21.15.106:0/2098723210' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-10T00:51:46.774 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:51:46 smithi170 ceph-mon[27620]: pgmap v194: 1 pgs: 1 active+clean; 577 KiB data, 229 MiB used, 715 GiB / 715 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-03-10T00:51:46.774 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:51:46 smithi170 ceph-mon[27620]: from='client.? 172.21.15.106:0/2098723210' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-10T00:51:46.911 INFO:tasks.cephadm:Adding ceph.rgw.foo.a on smithi106 2024-03-10T00:51:46.911 DEBUG:teuthology.orchestra.run.smithi170:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:831bc188fa69f143bbba69ac76119e784500e8e3 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 75c37498-de77-11ee-95c7-87774f69a715 -- ceph orch apply rgw foo.a --placement '1;smithi106=foo.a' 2024-03-10T00:51:47.161 INFO:teuthology.orchestra.run.smithi170.stderr:Inferring config /var/lib/ceph/75c37498-de77-11ee-95c7-87774f69a715/mon.b/config 2024-03-10T00:51:48.589 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:51:48 smithi106 ceph-mon[25742]: pgmap v195: 1 pgs: 1 active+clean; 577 KiB data, 229 MiB used, 715 GiB / 715 GiB avail; 75 KiB/s, 0 objects/s recovering 2024-03-10T00:51:48.589 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:51:48 smithi106 ceph-mon[33460]: pgmap v195: 1 pgs: 1 active+clean; 577 KiB data, 229 MiB used, 715 GiB / 715 GiB avail; 75 KiB/s, 0 objects/s recovering 2024-03-10T00:51:48.774 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:51:48 smithi170 ceph-mon[27620]: pgmap v195: 1 pgs: 1 active+clean; 577 KiB data, 229 MiB used, 715 GiB / 715 GiB avail; 75 KiB/s, 0 objects/s recovering 2024-03-10T00:51:48.933 INFO:teuthology.orchestra.run.smithi170.stdout:Scheduled rgw.foo.a update... 2024-03-10T00:51:49.530 DEBUG:teuthology.orchestra.run.smithi106:rgw.foo.a> sudo journalctl -f -n 0 -u ceph-75c37498-de77-11ee-95c7-87774f69a715@rgw.foo.a.service 2024-03-10T00:51:49.532 INFO:tasks.cephadm:Adding ceph.iscsi.iscsi.a on smithi170 2024-03-10T00:51:49.532 DEBUG:teuthology.orchestra.run.smithi170:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:831bc188fa69f143bbba69ac76119e784500e8e3 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 75c37498-de77-11ee-95c7-87774f69a715 -- ceph osd pool create datapool 3 3 replicated 2024-03-10T00:51:49.770 INFO:teuthology.orchestra.run.smithi170.stderr:Inferring config /var/lib/ceph/75c37498-de77-11ee-95c7-87774f69a715/mon.b/config 2024-03-10T00:51:50.088 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:51:49 smithi106 ceph-mon[33460]: from='client.24307 -' entity='client.admin' cmd=[{"prefix": "orch apply rgw", "svc_id": "foo.a", "placement": "1;smithi106=foo.a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-10T00:51:50.089 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:51:49 smithi106 ceph-mon[33460]: Saving service rgw.foo.a spec with placement smithi106=foo.a;count:1 2024-03-10T00:51:50.089 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:51:49 smithi106 ceph-mon[33460]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:51:50.089 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:51:49 smithi106 ceph-mon[33460]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-10T00:51:50.089 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:51:49 smithi106 ceph-mon[33460]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-10T00:51:50.089 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:51:49 smithi106 ceph-mon[33460]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-10T00:51:50.089 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:51:49 smithi106 ceph-mon[33460]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:51:50.089 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:51:49 smithi106 ceph-mon[33460]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.a", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-03-10T00:51:50.089 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:51:49 smithi106 ceph-mon[33460]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.a", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-03-10T00:51:50.089 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:51:49 smithi106 ceph-mon[33460]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:51:50.089 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:51:49 smithi106 ceph-mon[33460]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-10T00:51:50.089 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:51:49 smithi106 ceph-mon[25742]: from='client.24307 -' entity='client.admin' cmd=[{"prefix": "orch apply rgw", "svc_id": "foo.a", "placement": "1;smithi106=foo.a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-10T00:51:50.090 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:51:49 smithi106 ceph-mon[25742]: Saving service rgw.foo.a spec with placement smithi106=foo.a;count:1 2024-03-10T00:51:50.090 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:51:49 smithi106 ceph-mon[25742]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:51:50.090 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:51:49 smithi106 ceph-mon[25742]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-10T00:51:50.090 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:51:49 smithi106 ceph-mon[25742]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-10T00:51:50.090 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:51:49 smithi106 ceph-mon[25742]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-10T00:51:50.090 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:51:49 smithi106 ceph-mon[25742]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:51:50.090 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:51:49 smithi106 ceph-mon[25742]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.a", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-03-10T00:51:50.090 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:51:49 smithi106 ceph-mon[25742]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.a", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-03-10T00:51:50.090 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:51:49 smithi106 ceph-mon[25742]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:51:50.090 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:51:49 smithi106 ceph-mon[25742]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-10T00:51:50.164 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:51:49 smithi170 ceph-mon[27620]: from='client.24307 -' entity='client.admin' cmd=[{"prefix": "orch apply rgw", "svc_id": "foo.a", "placement": "1;smithi106=foo.a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-10T00:51:50.164 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:51:49 smithi170 ceph-mon[27620]: Saving service rgw.foo.a spec with placement smithi106=foo.a;count:1 2024-03-10T00:51:50.164 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:51:49 smithi170 ceph-mon[27620]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:51:50.164 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:51:49 smithi170 ceph-mon[27620]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-10T00:51:50.164 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:51:49 smithi170 ceph-mon[27620]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-10T00:51:50.164 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:51:49 smithi170 ceph-mon[27620]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-10T00:51:50.164 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:51:49 smithi170 ceph-mon[27620]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:51:50.164 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:51:49 smithi170 ceph-mon[27620]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.a", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-03-10T00:51:50.165 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:51:49 smithi170 ceph-mon[27620]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.a", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-03-10T00:51:50.165 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:51:49 smithi170 ceph-mon[27620]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:51:50.165 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:51:49 smithi170 ceph-mon[27620]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-10T00:51:51.026 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:51:50 smithi106 ceph-mon[25742]: Deploying daemon rgw.foo.a on smithi106 2024-03-10T00:51:51.026 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:51:50 smithi106 ceph-mon[25742]: pgmap v196: 1 pgs: 1 active+clean; 577 KiB data, 229 MiB used, 715 GiB / 715 GiB avail; 62 KiB/s, 0 objects/s recovering 2024-03-10T00:51:51.026 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:51:50 smithi106 ceph-mon[33460]: Deploying daemon rgw.foo.a on smithi106 2024-03-10T00:51:51.026 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:51:50 smithi106 ceph-mon[33460]: pgmap v196: 1 pgs: 1 active+clean; 577 KiB data, 229 MiB used, 715 GiB / 715 GiB avail; 62 KiB/s, 0 objects/s recovering 2024-03-10T00:51:51.172 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:51:50 smithi170 ceph-mon[27620]: Deploying daemon rgw.foo.a on smithi106 2024-03-10T00:51:51.172 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:51:50 smithi170 ceph-mon[27620]: pgmap v196: 1 pgs: 1 active+clean; 577 KiB data, 229 MiB used, 715 GiB / 715 GiB avail; 62 KiB/s, 0 objects/s recovering 2024-03-10T00:51:51.338 INFO:journalctl@ceph.rgw.foo.a.smithi106.stdout:Mar 10 00:51:51 smithi106 systemd[1]: Starting Ceph rgw.foo.a for 75c37498-de77-11ee-95c7-87774f69a715... 2024-03-10T00:51:51.688 INFO:journalctl@ceph.rgw.foo.a.smithi106.stdout:Mar 10 00:51:51 smithi106 podman[54924]: 2024-03-10 00:51:51.33882845 +0000 UTC m=+0.018007280 image pull 01a4f6a6bb3e92ab9ab39a2f835684ea9b10e6ec9fb81ac08ebea4165583a538 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bc4a24690ce5d7255d490e21058d8c114b55e1b93a3d8345f90b6700ad29dada 2024-03-10T00:51:51.688 INFO:journalctl@ceph.rgw.foo.a.smithi106.stdout:Mar 10 00:51:51 smithi106 podman[54924]: 2024-03-10 00:51:51.446583735 +0000 UTC m=+0.125762565 container create d9af1c13f4609b8d86c21d60d00fc35f34453e04bf1b09dd4ea1162b95dbaa4f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bc4a24690ce5d7255d490e21058d8c114b55e1b93a3d8345f90b6700ad29dada, name=ceph-75c37498-de77-11ee-95c7-87774f69a715-rgw-foo-a, org.label-schema.build-date=20240222, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, RELEASE=wip-yuri4-testing-2024-03-05-0854-831bc18, GIT_BRANCH=HEAD, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, ceph=True) 2024-03-10T00:51:51.688 INFO:journalctl@ceph.rgw.foo.a.smithi106.stdout:Mar 10 00:51:51 smithi106 podman[54924]: 2024-03-10 00:51:51.690508352 +0000 UTC m=+0.369687182 container init d9af1c13f4609b8d86c21d60d00fc35f34453e04bf1b09dd4ea1162b95dbaa4f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bc4a24690ce5d7255d490e21058d8c114b55e1b93a3d8345f90b6700ad29dada, name=ceph-75c37498-de77-11ee-95c7-87774f69a715-rgw-foo-a, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, RELEASE=wip-yuri4-testing-2024-03-05-0854-831bc18, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, org.label-schema.build-date=20240222, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_BRANCH=HEAD) 2024-03-10T00:51:52.088 INFO:journalctl@ceph.rgw.foo.a.smithi106.stdout:Mar 10 00:51:51 smithi106 podman[54924]: 2024-03-10 00:51:51.695934139 +0000 UTC m=+0.375112977 container start d9af1c13f4609b8d86c21d60d00fc35f34453e04bf1b09dd4ea1162b95dbaa4f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bc4a24690ce5d7255d490e21058d8c114b55e1b93a3d8345f90b6700ad29dada, name=ceph-75c37498-de77-11ee-95c7-87774f69a715-rgw-foo-a, org.label-schema.build-date=20240222, io.buildah.version=1.33.5, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, RELEASE=wip-yuri4-testing-2024-03-05-0854-831bc18, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0) 2024-03-10T00:51:52.088 INFO:journalctl@ceph.rgw.foo.a.smithi106.stdout:Mar 10 00:51:51 smithi106 bash[54924]: d9af1c13f4609b8d86c21d60d00fc35f34453e04bf1b09dd4ea1162b95dbaa4f 2024-03-10T00:51:52.088 INFO:journalctl@ceph.rgw.foo.a.smithi106.stdout:Mar 10 00:51:51 smithi106 systemd[1]: Started Ceph rgw.foo.a for 75c37498-de77-11ee-95c7-87774f69a715. 2024-03-10T00:51:52.098 INFO:teuthology.orchestra.run.smithi170.stderr:pool 'datapool' created 2024-03-10T00:51:52.467 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:51:52 smithi170 ceph-mon[27620]: from='client.? 172.21.15.170:0/4162951919' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]: dispatch 2024-03-10T00:51:52.468 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:51:52 smithi170 ceph-mon[27620]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]: dispatch 2024-03-10T00:51:52.588 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:51:52 smithi106 ceph-mon[25742]: from='client.? 172.21.15.170:0/4162951919' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]: dispatch 2024-03-10T00:51:52.588 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:51:52 smithi106 ceph-mon[25742]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]: dispatch 2024-03-10T00:51:52.588 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:51:52 smithi106 ceph-mon[33460]: from='client.? 172.21.15.170:0/4162951919' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]: dispatch 2024-03-10T00:51:52.588 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:51:52 smithi106 ceph-mon[33460]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]: dispatch 2024-03-10T00:51:52.878 DEBUG:teuthology.orchestra.run.smithi170:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:831bc188fa69f143bbba69ac76119e784500e8e3 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 75c37498-de77-11ee-95c7-87774f69a715 -- rbd pool init datapool 2024-03-10T00:51:53.130 INFO:teuthology.orchestra.run.smithi170.stderr:Inferring config /var/lib/ceph/75c37498-de77-11ee-95c7-87774f69a715/mon.b/config 2024-03-10T00:51:53.253 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:51:53 smithi170 ceph-mon[27620]: pgmap v197: 1 pgs: 1 active+clean; 577 KiB data, 229 MiB used, 715 GiB / 715 GiB avail; 56 KiB/s, 0 objects/s recovering 2024-03-10T00:51:53.253 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:51:53 smithi170 ceph-mon[27620]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]': finished 2024-03-10T00:51:53.253 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:51:53 smithi170 ceph-mon[27620]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:51:53.253 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:51:53 smithi170 ceph-mon[27620]: osdmap e59: 8 total, 8 up, 8 in 2024-03-10T00:51:53.253 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:51:53 smithi170 ceph-mon[27620]: from='client.? 172.21.15.106:0/3377605471' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]: dispatch 2024-03-10T00:51:53.253 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:51:53 smithi170 ceph-mon[27620]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:51:53.253 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:51:53 smithi170 ceph-mon[27620]: from='client.? ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]: dispatch 2024-03-10T00:51:53.253 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:51:53 smithi170 ceph-mon[27620]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:51:53.253 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:51:53 smithi170 ceph-mon[27620]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:51:53.254 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:51:53 smithi170 ceph-mon[27620]: Detected new or changed devices on smithi170 2024-03-10T00:51:53.254 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:51:53 smithi170 ceph-mon[27620]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:51:53.254 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:51:53 smithi170 ceph-mon[27620]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:51:53.254 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:51:53 smithi170 ceph-mon[27620]: Saving service rgw.foo.a spec with placement smithi106=foo.a;count:1 2024-03-10T00:51:53.254 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:51:53 smithi170 ceph-mon[27620]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:51:53.254 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:51:53 smithi170 ceph-mon[27620]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:51:53.254 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:51:53 smithi170 ceph-mon[27620]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-10T00:51:53.254 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:51:53 smithi170 ceph-mon[27620]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-10T00:51:53.254 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:51:53 smithi170 ceph-mon[27620]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-10T00:51:53.254 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:51:53 smithi170 ceph-mon[27620]: Checking dashboard <-> RGW credentials 2024-03-10T00:51:53.321 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:51:53 smithi106 ceph-mon[25742]: pgmap v197: 1 pgs: 1 active+clean; 577 KiB data, 229 MiB used, 715 GiB / 715 GiB avail; 56 KiB/s, 0 objects/s recovering 2024-03-10T00:51:53.321 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:51:53 smithi106 ceph-mon[25742]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]': finished 2024-03-10T00:51:53.321 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:51:53 smithi106 ceph-mon[25742]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:51:53.321 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:51:53 smithi106 ceph-mon[25742]: osdmap e59: 8 total, 8 up, 8 in 2024-03-10T00:51:53.322 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:51:53 smithi106 ceph-mon[25742]: from='client.? 172.21.15.106:0/3377605471' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]: dispatch 2024-03-10T00:51:53.322 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:51:53 smithi106 ceph-mon[25742]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:51:53.322 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:51:53 smithi106 ceph-mon[25742]: from='client.? ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]: dispatch 2024-03-10T00:51:53.322 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:51:53 smithi106 ceph-mon[25742]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:51:53.322 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:51:53 smithi106 ceph-mon[25742]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:51:53.322 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:51:53 smithi106 ceph-mon[25742]: Detected new or changed devices on smithi170 2024-03-10T00:51:53.322 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:51:53 smithi106 ceph-mon[25742]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:51:53.322 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:51:53 smithi106 ceph-mon[25742]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:51:53.322 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:51:53 smithi106 ceph-mon[25742]: Saving service rgw.foo.a spec with placement smithi106=foo.a;count:1 2024-03-10T00:51:53.322 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:51:53 smithi106 ceph-mon[25742]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:51:53.322 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:51:53 smithi106 ceph-mon[25742]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:51:53.323 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:51:53 smithi106 ceph-mon[25742]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-10T00:51:53.323 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:51:53 smithi106 ceph-mon[25742]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-10T00:51:53.323 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:51:53 smithi106 ceph-mon[25742]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-10T00:51:53.323 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:51:53 smithi106 ceph-mon[25742]: Checking dashboard <-> RGW credentials 2024-03-10T00:51:53.323 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:51:53 smithi106 ceph-mon[33460]: pgmap v197: 1 pgs: 1 active+clean; 577 KiB data, 229 MiB used, 715 GiB / 715 GiB avail; 56 KiB/s, 0 objects/s recovering 2024-03-10T00:51:53.323 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:51:53 smithi106 ceph-mon[33460]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]': finished 2024-03-10T00:51:53.323 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:51:53 smithi106 ceph-mon[33460]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:51:53.323 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:51:53 smithi106 ceph-mon[33460]: osdmap e59: 8 total, 8 up, 8 in 2024-03-10T00:51:53.323 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:51:53 smithi106 ceph-mon[33460]: from='client.? 172.21.15.106:0/3377605471' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]: dispatch 2024-03-10T00:51:53.324 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:51:53 smithi106 ceph-mon[33460]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:51:53.324 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:51:53 smithi106 ceph-mon[33460]: from='client.? ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]: dispatch 2024-03-10T00:51:53.324 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:51:53 smithi106 ceph-mon[33460]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:51:53.324 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:51:53 smithi106 ceph-mon[33460]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:51:53.324 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:51:53 smithi106 ceph-mon[33460]: Detected new or changed devices on smithi170 2024-03-10T00:51:53.324 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:51:53 smithi106 ceph-mon[33460]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:51:53.324 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:51:53 smithi106 ceph-mon[33460]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:51:53.324 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:51:53 smithi106 ceph-mon[33460]: Saving service rgw.foo.a spec with placement smithi106=foo.a;count:1 2024-03-10T00:51:53.324 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:51:53 smithi106 ceph-mon[33460]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:51:53.324 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:51:53 smithi106 ceph-mon[33460]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:51:53.324 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:51:53 smithi106 ceph-mon[33460]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-10T00:51:53.324 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:51:53 smithi106 ceph-mon[33460]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-10T00:51:53.324 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:51:53 smithi106 ceph-mon[33460]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-10T00:51:53.325 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:51:53 smithi106 ceph-mon[33460]: Checking dashboard <-> RGW credentials 2024-03-10T00:51:54.130 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:51:54 smithi170 ceph-mon[27620]: from='client.? ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]': finished 2024-03-10T00:51:54.230 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:51:54 smithi170 ceph-mon[27620]: osdmap e60: 8 total, 8 up, 8 in 2024-03-10T00:51:54.230 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:51:54 smithi170 ceph-mon[27620]: pgmap v200: 36 pgs: 11 creating+peering, 18 unknown, 7 active+clean; 577 KiB data, 229 MiB used, 715 GiB / 715 GiB avail; 127 B/s wr, 0 op/s 2024-03-10T00:51:54.588 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:51:54 smithi106 ceph-mon[25742]: from='client.? ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]': finished 2024-03-10T00:51:54.588 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:51:54 smithi106 ceph-mon[25742]: osdmap e60: 8 total, 8 up, 8 in 2024-03-10T00:51:54.588 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:51:54 smithi106 ceph-mon[25742]: pgmap v200: 36 pgs: 11 creating+peering, 18 unknown, 7 active+clean; 577 KiB data, 229 MiB used, 715 GiB / 715 GiB avail; 127 B/s wr, 0 op/s 2024-03-10T00:51:54.589 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:51:54 smithi106 ceph-mon[33460]: from='client.? ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]': finished 2024-03-10T00:51:54.589 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:51:54 smithi106 ceph-mon[33460]: osdmap e60: 8 total, 8 up, 8 in 2024-03-10T00:51:54.589 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:51:54 smithi106 ceph-mon[33460]: pgmap v200: 36 pgs: 11 creating+peering, 18 unknown, 7 active+clean; 577 KiB data, 229 MiB used, 715 GiB / 715 GiB avail; 127 B/s wr, 0 op/s 2024-03-10T00:51:55.338 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:51:55 smithi106 ceph-mon[33460]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-10T00:51:55.338 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:51:55 smithi106 ceph-mon[33460]: osdmap e61: 8 total, 8 up, 8 in 2024-03-10T00:51:55.338 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:51:55 smithi106 ceph-mon[33460]: from='client.? 172.21.15.106:0/3328248610' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-03-10T00:51:55.338 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:51:55 smithi106 ceph-mon[33460]: from='client.? 172.21.15.106:0/1521705161' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-03-10T00:51:55.338 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:51:55 smithi106 ceph-mon[33460]: from='client.? ' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-03-10T00:51:55.338 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:51:55 smithi106 ceph-mon[33460]: from='client.? 172.21.15.170:0/1508197259' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "datapool","app": "rbd"}]: dispatch 2024-03-10T00:51:55.338 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:51:55 smithi106 ceph-mon[33460]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "datapool","app": "rbd"}]: dispatch 2024-03-10T00:51:55.339 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:51:55 smithi106 ceph-mon[25742]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-10T00:51:55.339 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:51:55 smithi106 ceph-mon[25742]: osdmap e61: 8 total, 8 up, 8 in 2024-03-10T00:51:55.339 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:51:55 smithi106 ceph-mon[25742]: from='client.? 172.21.15.106:0/3328248610' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-03-10T00:51:55.339 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:51:55 smithi106 ceph-mon[25742]: from='client.? 172.21.15.106:0/1521705161' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-03-10T00:51:55.339 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:51:55 smithi106 ceph-mon[25742]: from='client.? ' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-03-10T00:51:55.339 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:51:55 smithi106 ceph-mon[25742]: from='client.? 172.21.15.170:0/1508197259' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "datapool","app": "rbd"}]: dispatch 2024-03-10T00:51:55.339 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:51:55 smithi106 ceph-mon[25742]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "datapool","app": "rbd"}]: dispatch 2024-03-10T00:51:55.482 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:51:55 smithi170 ceph-mon[27620]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-10T00:51:55.483 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:51:55 smithi170 ceph-mon[27620]: osdmap e61: 8 total, 8 up, 8 in 2024-03-10T00:51:55.483 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:51:55 smithi170 ceph-mon[27620]: from='client.? 172.21.15.106:0/3328248610' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-03-10T00:51:55.483 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:51:55 smithi170 ceph-mon[27620]: from='client.? 172.21.15.106:0/1521705161' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-03-10T00:51:55.483 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:51:55 smithi170 ceph-mon[27620]: from='client.? ' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-03-10T00:51:55.483 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:51:55 smithi170 ceph-mon[27620]: from='client.? 172.21.15.170:0/1508197259' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "datapool","app": "rbd"}]: dispatch 2024-03-10T00:51:55.483 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:51:55 smithi170 ceph-mon[27620]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "datapool","app": "rbd"}]: dispatch 2024-03-10T00:51:56.447 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:51:56 smithi106 ceph-mon[25742]: from='client.? 172.21.15.106:0/1521705161' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]': finished 2024-03-10T00:51:56.448 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:51:56 smithi106 ceph-mon[25742]: from='client.? ' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]': finished 2024-03-10T00:51:56.448 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:51:56 smithi106 ceph-mon[25742]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "datapool","app": "rbd"}]': finished 2024-03-10T00:51:56.448 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:51:56 smithi106 ceph-mon[25742]: osdmap e62: 8 total, 8 up, 8 in 2024-03-10T00:51:56.448 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:51:56 smithi106 ceph-mon[25742]: pgmap v203: 68 pgs: 20 creating+peering, 24 unknown, 24 active+clean; 577 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 767 B/s wr, 1 op/s 2024-03-10T00:51:56.448 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:51:56 smithi106 ceph-mon[25742]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:51:56.448 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:51:56 smithi106 ceph-mon[25742]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:51:56.448 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:51:56 smithi106 ceph-mon[33460]: from='client.? 172.21.15.106:0/1521705161' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]': finished 2024-03-10T00:51:56.448 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:51:56 smithi106 ceph-mon[33460]: from='client.? ' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]': finished 2024-03-10T00:51:56.448 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:51:56 smithi106 ceph-mon[33460]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "datapool","app": "rbd"}]': finished 2024-03-10T00:51:56.448 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:51:56 smithi106 ceph-mon[33460]: osdmap e62: 8 total, 8 up, 8 in 2024-03-10T00:51:56.449 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:51:56 smithi106 ceph-mon[33460]: pgmap v203: 68 pgs: 20 creating+peering, 24 unknown, 24 active+clean; 577 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 767 B/s wr, 1 op/s 2024-03-10T00:51:56.449 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:51:56 smithi106 ceph-mon[33460]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:51:56.449 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:51:56 smithi106 ceph-mon[33460]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:51:56.524 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:51:56 smithi170 ceph-mon[27620]: from='client.? 172.21.15.106:0/1521705161' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]': finished 2024-03-10T00:51:56.524 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:51:56 smithi170 ceph-mon[27620]: from='client.? ' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]': finished 2024-03-10T00:51:56.524 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:51:56 smithi170 ceph-mon[27620]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "datapool","app": "rbd"}]': finished 2024-03-10T00:51:56.524 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:51:56 smithi170 ceph-mon[27620]: osdmap e62: 8 total, 8 up, 8 in 2024-03-10T00:51:56.525 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:51:56 smithi170 ceph-mon[27620]: pgmap v203: 68 pgs: 20 creating+peering, 24 unknown, 24 active+clean; 577 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 767 B/s wr, 1 op/s 2024-03-10T00:51:56.525 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:51:56 smithi170 ceph-mon[27620]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:51:56.525 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:51:56 smithi170 ceph-mon[27620]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:51:57.473 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:51:57 smithi170 ceph-mon[27620]: osdmap e63: 8 total, 8 up, 8 in 2024-03-10T00:51:57.474 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:51:57 smithi170 ceph-mon[27620]: from='client.? 172.21.15.106:0/3328248610' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-03-10T00:51:57.474 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:51:57 smithi170 ceph-mon[27620]: from='client.? 172.21.15.106:0/1521705161' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-03-10T00:51:57.474 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:51:57 smithi170 ceph-mon[27620]: from='client.? ' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-03-10T00:51:57.474 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:51:57 smithi170 ceph-mon[27620]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-03-10T00:51:57.474 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:51:57 smithi170 ceph-mon[27620]: Cluster is now healthy 2024-03-10T00:51:57.474 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:51:57 smithi170 ceph-mon[27620]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:51:57.475 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:51:57 smithi170 ceph-mon[27620]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:51:57.475 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:51:57 smithi170 ceph-mon[27620]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:51:57.475 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:51:57 smithi170 ceph-mon[27620]: from='client.? 172.21.15.106:0/1521705161' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]': finished 2024-03-10T00:51:57.475 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:51:57 smithi170 ceph-mon[27620]: from='client.? ' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]': finished 2024-03-10T00:51:57.475 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:51:57 smithi170 ceph-mon[27620]: osdmap e64: 8 total, 8 up, 8 in 2024-03-10T00:51:57.588 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:51:57 smithi106 ceph-mon[33460]: osdmap e63: 8 total, 8 up, 8 in 2024-03-10T00:51:57.588 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:51:57 smithi106 ceph-mon[33460]: from='client.? 172.21.15.106:0/3328248610' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-03-10T00:51:57.588 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:51:57 smithi106 ceph-mon[33460]: from='client.? 172.21.15.106:0/1521705161' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-03-10T00:51:57.588 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:51:57 smithi106 ceph-mon[33460]: from='client.? ' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-03-10T00:51:57.589 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:51:57 smithi106 ceph-mon[33460]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-03-10T00:51:57.589 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:51:57 smithi106 ceph-mon[33460]: Cluster is now healthy 2024-03-10T00:51:57.589 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:51:57 smithi106 ceph-mon[33460]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:51:57.589 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:51:57 smithi106 ceph-mon[33460]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:51:57.589 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:51:57 smithi106 ceph-mon[33460]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:51:57.589 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:51:57 smithi106 ceph-mon[33460]: from='client.? 172.21.15.106:0/1521705161' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]': finished 2024-03-10T00:51:57.589 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:51:57 smithi106 ceph-mon[33460]: from='client.? ' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]': finished 2024-03-10T00:51:57.589 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:51:57 smithi106 ceph-mon[33460]: osdmap e64: 8 total, 8 up, 8 in 2024-03-10T00:51:57.589 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:51:57 smithi106 ceph-mon[25742]: osdmap e63: 8 total, 8 up, 8 in 2024-03-10T00:51:57.589 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:51:57 smithi106 ceph-mon[25742]: from='client.? 172.21.15.106:0/3328248610' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-03-10T00:51:57.590 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:51:57 smithi106 ceph-mon[25742]: from='client.? 172.21.15.106:0/1521705161' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-03-10T00:51:57.590 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:51:57 smithi106 ceph-mon[25742]: from='client.? ' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-03-10T00:51:57.590 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:51:57 smithi106 ceph-mon[25742]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-03-10T00:51:57.590 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:51:57 smithi106 ceph-mon[25742]: Cluster is now healthy 2024-03-10T00:51:57.590 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:51:57 smithi106 ceph-mon[25742]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:51:57.590 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:51:57 smithi106 ceph-mon[25742]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:51:57.590 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:51:57 smithi106 ceph-mon[25742]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:51:57.590 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:51:57 smithi106 ceph-mon[25742]: from='client.? 172.21.15.106:0/1521705161' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]': finished 2024-03-10T00:51:57.590 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:51:57 smithi106 ceph-mon[25742]: from='client.? ' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]': finished 2024-03-10T00:51:57.590 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:51:57 smithi106 ceph-mon[25742]: osdmap e64: 8 total, 8 up, 8 in 2024-03-10T00:51:58.106 DEBUG:teuthology.orchestra.run.smithi170:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:831bc188fa69f143bbba69ac76119e784500e8e3 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 75c37498-de77-11ee-95c7-87774f69a715 -- ceph orch apply iscsi datapool admin admin --trusted_ip_list 172.21.15.170 --placement '1;smithi170=iscsi.a' 2024-03-10T00:51:58.352 INFO:teuthology.orchestra.run.smithi170.stderr:Inferring config /var/lib/ceph/75c37498-de77-11ee-95c7-87774f69a715/mon.b/config 2024-03-10T00:51:58.461 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:51:58 smithi170 ceph-mon[27620]: pgmap v206: 100 pgs: 20 creating+peering, 51 unknown, 29 active+clean; 578 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 2.0 KiB/s rd, 767 B/s wr, 3 op/s 2024-03-10T00:51:58.461 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:51:58 smithi170 ceph-mon[27620]: osdmap e65: 8 total, 8 up, 8 in 2024-03-10T00:51:58.461 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:51:58 smithi170 ceph-mon[27620]: from='client.? 172.21.15.106:0/1521705161' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-03-10T00:51:58.461 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:51:58 smithi170 ceph-mon[27620]: from='client.? 172.21.15.106:0/3328248610' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-03-10T00:51:58.461 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:51:58 smithi170 ceph-mon[27620]: from='client.? ' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-03-10T00:51:58.553 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:51:58 smithi106 ceph-mon[33460]: pgmap v206: 100 pgs: 20 creating+peering, 51 unknown, 29 active+clean; 578 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 2.0 KiB/s rd, 767 B/s wr, 3 op/s 2024-03-10T00:51:58.553 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:51:58 smithi106 ceph-mon[33460]: osdmap e65: 8 total, 8 up, 8 in 2024-03-10T00:51:58.553 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:51:58 smithi106 ceph-mon[33460]: from='client.? 172.21.15.106:0/1521705161' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-03-10T00:51:58.553 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:51:58 smithi106 ceph-mon[33460]: from='client.? 172.21.15.106:0/3328248610' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-03-10T00:51:58.553 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:51:58 smithi106 ceph-mon[33460]: from='client.? ' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-03-10T00:51:58.553 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:51:58 smithi106 ceph-mon[25742]: pgmap v206: 100 pgs: 20 creating+peering, 51 unknown, 29 active+clean; 578 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 2.0 KiB/s rd, 767 B/s wr, 3 op/s 2024-03-10T00:51:58.553 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:51:58 smithi106 ceph-mon[25742]: osdmap e65: 8 total, 8 up, 8 in 2024-03-10T00:51:58.553 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:51:58 smithi106 ceph-mon[25742]: from='client.? 172.21.15.106:0/1521705161' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-03-10T00:51:58.553 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:51:58 smithi106 ceph-mon[25742]: from='client.? 172.21.15.106:0/3328248610' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-03-10T00:51:58.554 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:51:58 smithi106 ceph-mon[25742]: from='client.? ' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-03-10T00:52:00.254 INFO:teuthology.orchestra.run.smithi170.stdout:Scheduled iscsi.datapool update... 2024-03-10T00:52:00.524 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:52:00 smithi170 ceph-mon[27620]: from='client.? 172.21.15.106:0/1521705161' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]': finished 2024-03-10T00:52:00.524 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:52:00 smithi170 ceph-mon[27620]: from='client.? ' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]': finished 2024-03-10T00:52:00.524 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:52:00 smithi170 ceph-mon[27620]: osdmap e66: 8 total, 8 up, 8 in 2024-03-10T00:52:00.524 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:52:00 smithi170 ceph-mon[27620]: from='client.? 172.21.15.106:0/1521705161' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-03-10T00:52:00.524 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:52:00 smithi170 ceph-mon[27620]: from='client.? 172.21.15.106:0/3328248610' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-03-10T00:52:00.524 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:52:00 smithi170 ceph-mon[27620]: from='client.? ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-03-10T00:52:00.525 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:52:00 smithi170 ceph-mon[27620]: pgmap v209: 132 pgs: 17 creating+peering, 41 unknown, 74 active+clean; 578 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 3.2 KiB/s rd, 1023 B/s wr, 5 op/s 2024-03-10T00:52:00.588 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:52:00 smithi106 ceph-mon[25742]: from='client.? 172.21.15.106:0/1521705161' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]': finished 2024-03-10T00:52:00.588 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:52:00 smithi106 ceph-mon[25742]: from='client.? ' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]': finished 2024-03-10T00:52:00.588 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:52:00 smithi106 ceph-mon[25742]: osdmap e66: 8 total, 8 up, 8 in 2024-03-10T00:52:00.588 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:52:00 smithi106 ceph-mon[25742]: from='client.? 172.21.15.106:0/1521705161' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-03-10T00:52:00.588 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:52:00 smithi106 ceph-mon[25742]: from='client.? 172.21.15.106:0/3328248610' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-03-10T00:52:00.589 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:52:00 smithi106 ceph-mon[25742]: from='client.? ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-03-10T00:52:00.589 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:52:00 smithi106 ceph-mon[25742]: pgmap v209: 132 pgs: 17 creating+peering, 41 unknown, 74 active+clean; 578 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 3.2 KiB/s rd, 1023 B/s wr, 5 op/s 2024-03-10T00:52:00.589 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:52:00 smithi106 ceph-mon[33460]: from='client.? 172.21.15.106:0/1521705161' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]': finished 2024-03-10T00:52:00.589 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:52:00 smithi106 ceph-mon[33460]: from='client.? ' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]': finished 2024-03-10T00:52:00.589 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:52:00 smithi106 ceph-mon[33460]: osdmap e66: 8 total, 8 up, 8 in 2024-03-10T00:52:00.589 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:52:00 smithi106 ceph-mon[33460]: from='client.? 172.21.15.106:0/1521705161' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-03-10T00:52:00.589 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:52:00 smithi106 ceph-mon[33460]: from='client.? 172.21.15.106:0/3328248610' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-03-10T00:52:00.589 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:52:00 smithi106 ceph-mon[33460]: from='client.? ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-03-10T00:52:00.589 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:52:00 smithi106 ceph-mon[33460]: pgmap v209: 132 pgs: 17 creating+peering, 41 unknown, 74 active+clean; 578 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 3.2 KiB/s rd, 1023 B/s wr, 5 op/s 2024-03-10T00:52:00.936 INFO:tasks.cephadm:Distributing iscsi-gateway.cfg... 2024-03-10T00:52:00.937 DEBUG:teuthology.orchestra.run.smithi106:> set -ex 2024-03-10T00:52:00.937 DEBUG:teuthology.orchestra.run.smithi106:> sudo dd of=/etc/ceph/iscsi-gateway.cfg 2024-03-10T00:52:00.968 DEBUG:teuthology.orchestra.run.smithi170:> set -ex 2024-03-10T00:52:00.969 DEBUG:teuthology.orchestra.run.smithi170:> sudo dd of=/etc/ceph/iscsi-gateway.cfg 2024-03-10T00:52:00.999 DEBUG:teuthology.orchestra.run.smithi170:iscsi.iscsi.a> sudo journalctl -f -n 0 -u ceph-75c37498-de77-11ee-95c7-87774f69a715@iscsi.iscsi.a.service 2024-03-10T00:52:01.042 INFO:tasks.cephadm:Adding prometheus.a on smithi170 2024-03-10T00:52:01.042 DEBUG:teuthology.orchestra.run.smithi170:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:831bc188fa69f143bbba69ac76119e784500e8e3 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 75c37498-de77-11ee-95c7-87774f69a715 -- ceph orch apply prometheus '1;smithi170=a' 2024-03-10T00:52:01.238 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:52:01 smithi170 ceph-mon[27620]: from='client.? 172.21.15.106:0/1521705161' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-03-10T00:52:01.239 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:52:01 smithi170 ceph-mon[27620]: from='client.? ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-03-10T00:52:01.239 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:52:01 smithi170 ceph-mon[27620]: osdmap e67: 8 total, 8 up, 8 in 2024-03-10T00:52:01.239 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:52:01 smithi170 ceph-mon[27620]: from='client.24370 -' entity='client.admin' cmd=[{"prefix": "orch apply iscsi", "pool": "datapool", "api_user": "admin", "api_password": "admin", "trusted_ip_list": "172.21.15.170", "placement": "1;smithi170=iscsi.a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-10T00:52:01.239 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:52:01 smithi170 ceph-mon[27620]: Saving service iscsi.datapool spec with placement smithi170=iscsi.a;count:1 2024-03-10T00:52:01.239 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:52:01 smithi170 ceph-mon[27620]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:52:01.239 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:52:01 smithi170 ceph-mon[27620]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:52:01.239 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:52:01 smithi170 ceph-mon[27620]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:52:01.239 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:52:01 smithi170 ceph-mon[27620]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:52:01.239 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:52:01 smithi170 ceph-mon[27620]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.iscsi.a", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-10T00:52:01.239 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:52:01 smithi170 ceph-mon[27620]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.iscsi.iscsi.a", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]': finished 2024-03-10T00:52:01.239 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:52:01 smithi170 ceph-mon[27620]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-10T00:52:01.239 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:52:01 smithi170 ceph-mon[27620]: Deploying daemon iscsi.iscsi.a on smithi170 2024-03-10T00:52:01.338 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:52:01 smithi106 ceph-mon[33460]: from='client.? 172.21.15.106:0/1521705161' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-03-10T00:52:01.338 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:52:01 smithi106 ceph-mon[33460]: from='client.? ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-03-10T00:52:01.339 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:52:01 smithi106 ceph-mon[33460]: osdmap e67: 8 total, 8 up, 8 in 2024-03-10T00:52:01.339 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:52:01 smithi106 ceph-mon[33460]: from='client.24370 -' entity='client.admin' cmd=[{"prefix": "orch apply iscsi", "pool": "datapool", "api_user": "admin", "api_password": "admin", "trusted_ip_list": "172.21.15.170", "placement": "1;smithi170=iscsi.a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-10T00:52:01.339 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:52:01 smithi106 ceph-mon[33460]: Saving service iscsi.datapool spec with placement smithi170=iscsi.a;count:1 2024-03-10T00:52:01.339 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:52:01 smithi106 ceph-mon[33460]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:52:01.339 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:52:01 smithi106 ceph-mon[33460]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:52:01.339 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:52:01 smithi106 ceph-mon[33460]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:52:01.339 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:52:01 smithi106 ceph-mon[33460]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:52:01.339 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:52:01 smithi106 ceph-mon[33460]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.iscsi.a", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-10T00:52:01.339 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:52:01 smithi106 ceph-mon[33460]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.iscsi.iscsi.a", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]': finished 2024-03-10T00:52:01.339 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:52:01 smithi106 ceph-mon[33460]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-10T00:52:01.339 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:52:01 smithi106 ceph-mon[33460]: Deploying daemon iscsi.iscsi.a on smithi170 2024-03-10T00:52:01.339 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:52:01 smithi106 ceph-mon[25742]: from='client.? 172.21.15.106:0/1521705161' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-03-10T00:52:01.340 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:52:01 smithi106 ceph-mon[25742]: from='client.? ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-03-10T00:52:01.340 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:52:01 smithi106 ceph-mon[25742]: osdmap e67: 8 total, 8 up, 8 in 2024-03-10T00:52:01.340 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:52:01 smithi106 ceph-mon[25742]: from='client.24370 -' entity='client.admin' cmd=[{"prefix": "orch apply iscsi", "pool": "datapool", "api_user": "admin", "api_password": "admin", "trusted_ip_list": "172.21.15.170", "placement": "1;smithi170=iscsi.a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-10T00:52:01.340 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:52:01 smithi106 ceph-mon[25742]: Saving service iscsi.datapool spec with placement smithi170=iscsi.a;count:1 2024-03-10T00:52:01.340 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:52:01 smithi106 ceph-mon[25742]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:52:01.340 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:52:01 smithi106 ceph-mon[25742]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:52:01.340 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:52:01 smithi106 ceph-mon[25742]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:52:01.340 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:52:01 smithi106 ceph-mon[25742]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:52:01.340 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:52:01 smithi106 ceph-mon[25742]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.iscsi.a", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-10T00:52:01.340 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:52:01 smithi106 ceph-mon[25742]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.iscsi.iscsi.a", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]': finished 2024-03-10T00:52:01.340 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:52:01 smithi106 ceph-mon[25742]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-10T00:52:01.340 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:52:01 smithi106 ceph-mon[25742]: Deploying daemon iscsi.iscsi.a on smithi170 2024-03-10T00:52:02.126 INFO:teuthology.orchestra.run.smithi170.stderr:Inferring config /var/lib/ceph/75c37498-de77-11ee-95c7-87774f69a715/mon.b/config 2024-03-10T00:52:02.374 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:52:02 smithi170 ceph-mon[27620]: pgmap v211: 132 pgs: 8 creating+peering, 11 unknown, 113 active+clean; 578 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 2.1 KiB/s wr, 41 op/s 2024-03-10T00:52:02.588 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:52:02 smithi106 ceph-mon[25742]: pgmap v211: 132 pgs: 8 creating+peering, 11 unknown, 113 active+clean; 578 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 2.1 KiB/s wr, 41 op/s 2024-03-10T00:52:02.588 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:52:02 smithi106 ceph-mon[33460]: pgmap v211: 132 pgs: 8 creating+peering, 11 unknown, 113 active+clean; 578 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 2.1 KiB/s wr, 41 op/s 2024-03-10T00:52:03.944 INFO:journalctl@ceph.iscsi.iscsi.a.smithi170.stdout:Mar 10 00:52:03 smithi170 podman[50065]: 2024-03-10 00:52:03.870234398 +0000 UTC m=+0.753839033 container create 8e6ee66d69a226ced52ef8f21ac185a1b42eb4a624c9c2d7e54f9757bfce2463 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bc4a24690ce5d7255d490e21058d8c114b55e1b93a3d8345f90b6700ad29dada, name=ceph-75c37498-de77-11ee-95c7-87774f69a715-iscsi-iscsi-a, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, RELEASE=wip-yuri4-testing-2024-03-05-0854-831bc18, org.label-schema.schema-version=1.0, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240222, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-10T00:52:04.278 INFO:teuthology.orchestra.run.smithi170.stdout:Scheduled prometheus update... 2024-03-10T00:52:04.471 INFO:journalctl@ceph.iscsi.iscsi.a.smithi170.stdout:Mar 10 00:52:04 smithi170 podman[50065]: 2024-03-10 00:52:04.206768702 +0000 UTC m=+1.090373337 container init 8e6ee66d69a226ced52ef8f21ac185a1b42eb4a624c9c2d7e54f9757bfce2463 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bc4a24690ce5d7255d490e21058d8c114b55e1b93a3d8345f90b6700ad29dada, name=ceph-75c37498-de77-11ee-95c7-87774f69a715-iscsi-iscsi-a, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240222, GIT_BRANCH=HEAD, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, maintainer=Guillaume Abrioux , GIT_CLEAN=True, RELEASE=wip-yuri4-testing-2024-03-05-0854-831bc18) 2024-03-10T00:52:04.472 INFO:journalctl@ceph.iscsi.iscsi.a.smithi170.stdout:Mar 10 00:52:04 smithi170 podman[50065]: 2024-03-10 00:52:04.212499823 +0000 UTC m=+1.096104459 container start 8e6ee66d69a226ced52ef8f21ac185a1b42eb4a624c9c2d7e54f9757bfce2463 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bc4a24690ce5d7255d490e21058d8c114b55e1b93a3d8345f90b6700ad29dada, name=ceph-75c37498-de77-11ee-95c7-87774f69a715-iscsi-iscsi-a, org.label-schema.license=GPLv2, org.label-schema.build-date=20240222, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, RELEASE=wip-yuri4-testing-2024-03-05-0854-831bc18, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True) 2024-03-10T00:52:04.472 INFO:journalctl@ceph.iscsi.iscsi.a.smithi170.stdout:Mar 10 00:52:04 smithi170 bash[50065]: 8e6ee66d69a226ced52ef8f21ac185a1b42eb4a624c9c2d7e54f9757bfce2463 2024-03-10T00:52:04.472 INFO:journalctl@ceph.iscsi.iscsi.a.smithi170.stdout:Mar 10 00:52:04 smithi170 systemd[1]: Started Ceph iscsi.iscsi.a for 75c37498-de77-11ee-95c7-87774f69a715. 2024-03-10T00:52:04.774 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:52:04 smithi170 ceph-mon[27620]: pgmap v212: 132 pgs: 3 creating+peering, 129 active+clean; 580 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 62 KiB/s rd, 3.8 KiB/s wr, 138 op/s 2024-03-10T00:52:04.774 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:52:04 smithi170 ceph-mon[27620]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:52:04.774 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:52:04 smithi170 ceph-mon[27620]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:52:04.774 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:52:04 smithi170 ceph-mon[27620]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:52:04.775 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:52:04 smithi170 ceph-mon[27620]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:52:04.775 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:52:04 smithi170 ceph-mon[27620]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:52:04.775 INFO:journalctl@ceph.iscsi.iscsi.a.smithi170.stdout:Mar 10 00:52:04 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-iscsi-iscsi-a[50134]: debug Started the configuration object watcher 2024-03-10T00:52:04.775 INFO:journalctl@ceph.iscsi.iscsi.a.smithi170.stdout:Mar 10 00:52:04 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-iscsi-iscsi-a[50134]: debug Checking for config object changes every 1s 2024-03-10T00:52:04.775 INFO:journalctl@ceph.iscsi.iscsi.a.smithi170.stdout:Mar 10 00:52:04 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-iscsi-iscsi-a[50134]: debug Processing osd blocklist entries for this node 2024-03-10T00:52:04.838 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:52:04 smithi106 ceph-mon[33460]: pgmap v212: 132 pgs: 3 creating+peering, 129 active+clean; 580 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 62 KiB/s rd, 3.8 KiB/s wr, 138 op/s 2024-03-10T00:52:04.838 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:52:04 smithi106 ceph-mon[33460]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:52:04.838 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:52:04 smithi106 ceph-mon[33460]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:52:04.838 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:52:04 smithi106 ceph-mon[33460]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:52:04.838 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:52:04 smithi106 ceph-mon[33460]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:52:04.838 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:52:04 smithi106 ceph-mon[33460]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:52:04.839 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:52:04 smithi106 ceph-mon[25742]: pgmap v212: 132 pgs: 3 creating+peering, 129 active+clean; 580 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 62 KiB/s rd, 3.8 KiB/s wr, 138 op/s 2024-03-10T00:52:04.839 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:52:04 smithi106 ceph-mon[25742]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:52:04.839 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:52:04 smithi106 ceph-mon[25742]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:52:04.839 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:52:04 smithi106 ceph-mon[25742]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:52:04.839 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:52:04 smithi106 ceph-mon[25742]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:52:04.839 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:52:04 smithi106 ceph-mon[25742]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:52:05.192 INFO:journalctl@ceph.iscsi.iscsi.a.smithi170.stdout:Mar 10 00:52:04 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-iscsi-iscsi-a[50134]: debug Reading the configuration object to update local LIO configuration 2024-03-10T00:52:05.192 INFO:journalctl@ceph.iscsi.iscsi.a.smithi170.stdout:Mar 10 00:52:04 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-iscsi-iscsi-a[50134]: debug Configuration does not have an entry for this host(smithi170.front.sepia.ceph.com) - nothing to define to LIO 2024-03-10T00:52:05.192 INFO:journalctl@ceph.iscsi.iscsi.a.smithi170.stdout:Mar 10 00:52:04 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-iscsi-iscsi-a[50134]: debug * Running on http://[::]:5000/ (Press CTRL+C to quit) 2024-03-10T00:52:05.220 DEBUG:teuthology.orchestra.run.smithi170:prometheus.a> sudo journalctl -f -n 0 -u ceph-75c37498-de77-11ee-95c7-87774f69a715@prometheus.a.service 2024-03-10T00:52:05.222 INFO:tasks.cephadm:Adding node-exporter.a on smithi106 2024-03-10T00:52:05.222 INFO:tasks.cephadm:Adding node-exporter.b on smithi170 2024-03-10T00:52:05.223 DEBUG:teuthology.orchestra.run.smithi170:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:831bc188fa69f143bbba69ac76119e784500e8e3 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 75c37498-de77-11ee-95c7-87774f69a715 -- ceph orch apply node-exporter '2;smithi106=a;smithi170=b' 2024-03-10T00:52:05.774 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:52:05 smithi170 ceph-mon[27620]: from='client.24391 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "prometheus", "placement": "1;smithi170=a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-10T00:52:05.775 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:52:05 smithi170 ceph-mon[27620]: Saving service prometheus spec with placement smithi170=a;count:1 2024-03-10T00:52:05.775 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:52:05 smithi170 ceph-mon[27620]: Checking pool "datapool" exists for service iscsi.datapool 2024-03-10T00:52:05.775 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:52:05 smithi170 ceph-mon[27620]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-10T00:52:05.775 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:52:05 smithi170 ceph-mon[27620]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-10T00:52:05.775 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:52:05 smithi170 ceph-mon[27620]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-10T00:52:05.775 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:52:05 smithi170 ceph-mon[27620]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:52:05.775 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:52:05 smithi170 ceph-mon[27620]: Metadata not up to date on all hosts. Skipping non agent specs 2024-03-10T00:52:05.775 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:52:05 smithi170 ceph-mon[27620]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "dashboard iscsi-gateway-list"}]: dispatch 2024-03-10T00:52:05.775 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:52:05 smithi170 ceph-mon[27620]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard iscsi-gateway-list"}]: dispatch 2024-03-10T00:52:05.775 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:52:05 smithi170 ceph-mon[27620]: Adding iSCSI gateway http://:@172.21.15.170:5000 to Dashboard 2024-03-10T00:52:05.775 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:52:05 smithi170 ceph-mon[27620]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "dashboard set-iscsi-api-ssl-verification", "value": "true"}]: dispatch 2024-03-10T00:52:05.776 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:52:05 smithi170 ceph-mon[27620]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-iscsi-api-ssl-verification", "value": "true"}]: dispatch 2024-03-10T00:52:05.776 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:52:05 smithi170 ceph-mon[27620]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:52:05.776 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:52:05 smithi170 ceph-mon[27620]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "dashboard iscsi-gateway-add", "name": "smithi170"}]: dispatch 2024-03-10T00:52:05.776 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:52:05 smithi170 ceph-mon[27620]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard iscsi-gateway-add", "name": "smithi170"}]: dispatch 2024-03-10T00:52:05.776 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:52:05 smithi170 ceph-mon[27620]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:52:05.776 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:52:05 smithi170 ceph-mon[27620]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-10T00:52:05.776 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:52:05 smithi170 ceph-mon[27620]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-10T00:52:05.776 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:52:05 smithi170 ceph-mon[27620]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-10T00:52:05.776 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:52:05 smithi170 ceph-mon[27620]: Checking dashboard <-> RGW credentials 2024-03-10T00:52:05.776 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:52:05 smithi170 ceph-mon[27620]: from='client.? 172.21.15.170:0/3362709646' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-10T00:52:05.776 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:52:05 smithi170 ceph-mon[27620]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:52:05.776 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:52:05 smithi170 ceph-mon[27620]: Metadata not up to date on all hosts. Skipping non agent specs 2024-03-10T00:52:05.776 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:52:05 smithi170 ceph-mon[27620]: osdmap e68: 8 total, 8 up, 8 in 2024-03-10T00:52:05.776 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:52:05 smithi170 ceph-mon[27620]: mgrmap e16: y(active, since 5m), standbys: x 2024-03-10T00:52:05.839 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:52:05 smithi106 ceph-mon[25742]: from='client.24391 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "prometheus", "placement": "1;smithi170=a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-10T00:52:05.839 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:52:05 smithi106 ceph-mon[25742]: Saving service prometheus spec with placement smithi170=a;count:1 2024-03-10T00:52:05.839 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:52:05 smithi106 ceph-mon[25742]: Checking pool "datapool" exists for service iscsi.datapool 2024-03-10T00:52:05.839 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:52:05 smithi106 ceph-mon[25742]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-10T00:52:05.839 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:52:05 smithi106 ceph-mon[25742]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-10T00:52:05.839 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:52:05 smithi106 ceph-mon[25742]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-10T00:52:05.839 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:52:05 smithi106 ceph-mon[25742]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:52:05.839 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:52:05 smithi106 ceph-mon[25742]: Metadata not up to date on all hosts. Skipping non agent specs 2024-03-10T00:52:05.839 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:52:05 smithi106 ceph-mon[25742]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "dashboard iscsi-gateway-list"}]: dispatch 2024-03-10T00:52:05.839 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:52:05 smithi106 ceph-mon[25742]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard iscsi-gateway-list"}]: dispatch 2024-03-10T00:52:05.839 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:52:05 smithi106 ceph-mon[25742]: Adding iSCSI gateway http://:@172.21.15.170:5000 to Dashboard 2024-03-10T00:52:05.839 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:52:05 smithi106 ceph-mon[25742]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "dashboard set-iscsi-api-ssl-verification", "value": "true"}]: dispatch 2024-03-10T00:52:05.840 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:52:05 smithi106 ceph-mon[25742]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-iscsi-api-ssl-verification", "value": "true"}]: dispatch 2024-03-10T00:52:05.840 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:52:05 smithi106 ceph-mon[25742]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:52:05.840 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:52:05 smithi106 ceph-mon[25742]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "dashboard iscsi-gateway-add", "name": "smithi170"}]: dispatch 2024-03-10T00:52:05.840 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:52:05 smithi106 ceph-mon[25742]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard iscsi-gateway-add", "name": "smithi170"}]: dispatch 2024-03-10T00:52:05.840 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:52:05 smithi106 ceph-mon[25742]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:52:05.840 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:52:05 smithi106 ceph-mon[25742]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-10T00:52:05.840 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:52:05 smithi106 ceph-mon[25742]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-10T00:52:05.840 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:52:05 smithi106 ceph-mon[25742]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-10T00:52:05.840 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:52:05 smithi106 ceph-mon[25742]: Checking dashboard <-> RGW credentials 2024-03-10T00:52:05.840 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:52:05 smithi106 ceph-mon[25742]: from='client.? 172.21.15.170:0/3362709646' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-10T00:52:05.840 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:52:05 smithi106 ceph-mon[25742]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:52:05.840 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:52:05 smithi106 ceph-mon[25742]: Metadata not up to date on all hosts. Skipping non agent specs 2024-03-10T00:52:05.840 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:52:05 smithi106 ceph-mon[25742]: osdmap e68: 8 total, 8 up, 8 in 2024-03-10T00:52:05.841 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:52:05 smithi106 ceph-mon[25742]: mgrmap e16: y(active, since 5m), standbys: x 2024-03-10T00:52:05.841 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:52:05 smithi106 ceph-mon[33460]: from='client.24391 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "prometheus", "placement": "1;smithi170=a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-10T00:52:05.841 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:52:05 smithi106 ceph-mon[33460]: Saving service prometheus spec with placement smithi170=a;count:1 2024-03-10T00:52:05.841 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:52:05 smithi106 ceph-mon[33460]: Checking pool "datapool" exists for service iscsi.datapool 2024-03-10T00:52:05.841 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:52:05 smithi106 ceph-mon[33460]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-10T00:52:05.841 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:52:05 smithi106 ceph-mon[33460]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-10T00:52:05.841 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:52:05 smithi106 ceph-mon[33460]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-10T00:52:05.841 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:52:05 smithi106 ceph-mon[33460]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:52:05.841 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:52:05 smithi106 ceph-mon[33460]: Metadata not up to date on all hosts. Skipping non agent specs 2024-03-10T00:52:05.841 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:52:05 smithi106 ceph-mon[33460]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "dashboard iscsi-gateway-list"}]: dispatch 2024-03-10T00:52:05.841 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:52:05 smithi106 ceph-mon[33460]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard iscsi-gateway-list"}]: dispatch 2024-03-10T00:52:05.842 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:52:05 smithi106 ceph-mon[33460]: Adding iSCSI gateway http://:@172.21.15.170:5000 to Dashboard 2024-03-10T00:52:05.842 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:52:05 smithi106 ceph-mon[33460]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "dashboard set-iscsi-api-ssl-verification", "value": "true"}]: dispatch 2024-03-10T00:52:05.842 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:52:05 smithi106 ceph-mon[33460]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-iscsi-api-ssl-verification", "value": "true"}]: dispatch 2024-03-10T00:52:05.842 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:52:05 smithi106 ceph-mon[33460]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:52:05.842 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:52:05 smithi106 ceph-mon[33460]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "dashboard iscsi-gateway-add", "name": "smithi170"}]: dispatch 2024-03-10T00:52:05.842 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:52:05 smithi106 ceph-mon[33460]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard iscsi-gateway-add", "name": "smithi170"}]: dispatch 2024-03-10T00:52:05.842 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:52:05 smithi106 ceph-mon[33460]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:52:05.842 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:52:05 smithi106 ceph-mon[33460]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-10T00:52:05.842 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:52:05 smithi106 ceph-mon[33460]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-10T00:52:05.842 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:52:05 smithi106 ceph-mon[33460]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-10T00:52:05.842 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:52:05 smithi106 ceph-mon[33460]: Checking dashboard <-> RGW credentials 2024-03-10T00:52:05.842 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:52:05 smithi106 ceph-mon[33460]: from='client.? 172.21.15.170:0/3362709646' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-10T00:52:05.842 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:52:05 smithi106 ceph-mon[33460]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:52:05.843 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:52:05 smithi106 ceph-mon[33460]: Metadata not up to date on all hosts. Skipping non agent specs 2024-03-10T00:52:05.843 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:52:05 smithi106 ceph-mon[33460]: osdmap e68: 8 total, 8 up, 8 in 2024-03-10T00:52:05.843 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:52:05 smithi106 ceph-mon[33460]: mgrmap e16: y(active, since 5m), standbys: x 2024-03-10T00:52:05.916 INFO:teuthology.orchestra.run.smithi170.stderr:Inferring config /var/lib/ceph/75c37498-de77-11ee-95c7-87774f69a715/mon.b/config 2024-03-10T00:52:07.024 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:52:06 smithi170 ceph-mon[27620]: pgmap v214: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 113 KiB/s rd, 5.8 KiB/s wr, 250 op/s 2024-03-10T00:52:07.025 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:52:06 smithi170 ceph-mon[27620]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:52:07.088 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:52:06 smithi106 ceph-mon[33460]: pgmap v214: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 113 KiB/s rd, 5.8 KiB/s wr, 250 op/s 2024-03-10T00:52:07.088 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:52:06 smithi106 ceph-mon[33460]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:52:07.089 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:52:06 smithi106 ceph-mon[25742]: pgmap v214: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 113 KiB/s rd, 5.8 KiB/s wr, 250 op/s 2024-03-10T00:52:07.089 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:52:06 smithi106 ceph-mon[25742]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:52:08.967 INFO:teuthology.orchestra.run.smithi170.stdout:Scheduled node-exporter update... 2024-03-10T00:52:08.988 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:52:08 smithi170 ceph-mon[27620]: pgmap v215: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 90 KiB/s rd, 4.6 KiB/s wr, 201 op/s 2024-03-10T00:52:09.088 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:52:08 smithi106 ceph-mon[25742]: pgmap v215: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 90 KiB/s rd, 4.6 KiB/s wr, 201 op/s 2024-03-10T00:52:09.089 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:52:08 smithi106 ceph-mon[33460]: pgmap v215: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 90 KiB/s rd, 4.6 KiB/s wr, 201 op/s 2024-03-10T00:52:09.613 DEBUG:teuthology.orchestra.run.smithi106:node-exporter.a> sudo journalctl -f -n 0 -u ceph-75c37498-de77-11ee-95c7-87774f69a715@node-exporter.a.service 2024-03-10T00:52:09.617 DEBUG:teuthology.orchestra.run.smithi170:node-exporter.b> sudo journalctl -f -n 0 -u ceph-75c37498-de77-11ee-95c7-87774f69a715@node-exporter.b.service 2024-03-10T00:52:09.619 INFO:tasks.cephadm:Adding alertmanager.a on smithi106 2024-03-10T00:52:09.619 DEBUG:teuthology.orchestra.run.smithi170:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:831bc188fa69f143bbba69ac76119e784500e8e3 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 75c37498-de77-11ee-95c7-87774f69a715 -- ceph orch apply alertmanager '1;smithi106=a' 2024-03-10T00:52:10.238 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:52:09 smithi170 ceph-mon[27620]: from='client.24457 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "node-exporter", "placement": "2;smithi106=a;smithi170=b", "target": ["mon-mgr", ""]}]: dispatch 2024-03-10T00:52:10.239 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:52:09 smithi170 ceph-mon[27620]: Saving service node-exporter spec with placement smithi106=a;smithi170=b;count:2 2024-03-10T00:52:10.239 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:52:09 smithi170 ceph-mon[27620]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:52:10.239 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:52:09 smithi170 ceph-mon[27620]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-10T00:52:10.239 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:52:09 smithi170 ceph-mon[27620]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-10T00:52:10.239 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:52:09 smithi170 ceph-mon[27620]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-10T00:52:10.239 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:52:09 smithi170 ceph-mon[27620]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:52:10.239 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:52:09 smithi170 ceph-mon[27620]: Metadata not up to date on all hosts. Skipping non agent specs 2024-03-10T00:52:10.239 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:52:09 smithi170 ceph-mon[27620]: pgmap v216: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 79 KiB/s rd, 3.9 KiB/s wr, 175 op/s 2024-03-10T00:52:10.338 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:52:09 smithi106 ceph-mon[25742]: from='client.24457 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "node-exporter", "placement": "2;smithi106=a;smithi170=b", "target": ["mon-mgr", ""]}]: dispatch 2024-03-10T00:52:10.338 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:52:09 smithi106 ceph-mon[25742]: Saving service node-exporter spec with placement smithi106=a;smithi170=b;count:2 2024-03-10T00:52:10.338 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:52:09 smithi106 ceph-mon[25742]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:52:10.338 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:52:09 smithi106 ceph-mon[25742]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-10T00:52:10.338 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:52:09 smithi106 ceph-mon[25742]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-10T00:52:10.338 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:52:09 smithi106 ceph-mon[25742]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-10T00:52:10.339 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:52:09 smithi106 ceph-mon[25742]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:52:10.339 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:52:09 smithi106 ceph-mon[25742]: Metadata not up to date on all hosts. Skipping non agent specs 2024-03-10T00:52:10.339 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:52:09 smithi106 ceph-mon[25742]: pgmap v216: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 79 KiB/s rd, 3.9 KiB/s wr, 175 op/s 2024-03-10T00:52:10.339 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:52:09 smithi106 ceph-mon[33460]: from='client.24457 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "node-exporter", "placement": "2;smithi106=a;smithi170=b", "target": ["mon-mgr", ""]}]: dispatch 2024-03-10T00:52:10.339 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:52:09 smithi106 ceph-mon[33460]: Saving service node-exporter spec with placement smithi106=a;smithi170=b;count:2 2024-03-10T00:52:10.339 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:52:09 smithi106 ceph-mon[33460]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:52:10.339 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:52:09 smithi106 ceph-mon[33460]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-10T00:52:10.339 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:52:09 smithi106 ceph-mon[33460]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-10T00:52:10.339 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:52:09 smithi106 ceph-mon[33460]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-10T00:52:10.339 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:52:09 smithi106 ceph-mon[33460]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:52:10.339 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:52:09 smithi106 ceph-mon[33460]: Metadata not up to date on all hosts. Skipping non agent specs 2024-03-10T00:52:10.339 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:52:09 smithi106 ceph-mon[33460]: pgmap v216: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 79 KiB/s rd, 3.9 KiB/s wr, 175 op/s 2024-03-10T00:52:10.487 INFO:teuthology.orchestra.run.smithi170.stderr:Inferring config /var/lib/ceph/75c37498-de77-11ee-95c7-87774f69a715/mon.b/config 2024-03-10T00:52:12.018 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:52:11 smithi170 ceph-mon[27620]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:52:12.018 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:52:11 smithi170 ceph-mon[27620]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:52:12.088 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:52:11 smithi106 ceph-mon[25742]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:52:12.088 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:52:11 smithi106 ceph-mon[25742]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:52:12.089 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:52:11 smithi106 ceph-mon[33460]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:52:12.089 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:52:11 smithi106 ceph-mon[33460]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:52:12.813 INFO:teuthology.orchestra.run.smithi170.stdout:Scheduled alertmanager update... 2024-03-10T00:52:13.088 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:52:12 smithi106 ceph-mon[25742]: pgmap v217: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 67 KiB/s rd, 3.2 KiB/s wr, 146 op/s 2024-03-10T00:52:13.088 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:52:12 smithi106 ceph-mon[25742]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:52:13.088 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:52:12 smithi106 ceph-mon[33460]: pgmap v217: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 67 KiB/s rd, 3.2 KiB/s wr, 146 op/s 2024-03-10T00:52:13.088 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:52:12 smithi106 ceph-mon[33460]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:52:13.184 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:52:12 smithi170 ceph-mon[27620]: pgmap v217: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 67 KiB/s rd, 3.2 KiB/s wr, 146 op/s 2024-03-10T00:52:13.185 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:52:12 smithi170 ceph-mon[27620]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:52:13.554 DEBUG:teuthology.orchestra.run.smithi106:alertmanager.a> sudo journalctl -f -n 0 -u ceph-75c37498-de77-11ee-95c7-87774f69a715@alertmanager.a.service 2024-03-10T00:52:13.557 INFO:tasks.cephadm:Adding grafana.a on smithi170 2024-03-10T00:52:13.557 DEBUG:teuthology.orchestra.run.smithi170:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:831bc188fa69f143bbba69ac76119e784500e8e3 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 75c37498-de77-11ee-95c7-87774f69a715 -- ceph orch apply grafana '1;smithi170=a' 2024-03-10T00:52:13.934 INFO:teuthology.orchestra.run.smithi170.stderr:Inferring config /var/lib/ceph/75c37498-de77-11ee-95c7-87774f69a715/mon.b/config 2024-03-10T00:52:14.088 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:52:13 smithi106 ceph-mon[25742]: from='client.24463 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "alertmanager", "placement": "1;smithi106=a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-10T00:52:14.088 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:52:13 smithi106 ceph-mon[25742]: Saving service alertmanager spec with placement smithi106=a;count:1 2024-03-10T00:52:14.088 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:52:13 smithi106 ceph-mon[25742]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-10T00:52:14.088 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:52:13 smithi106 ceph-mon[25742]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-10T00:52:14.089 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:52:13 smithi106 ceph-mon[25742]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-10T00:52:14.089 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:52:13 smithi106 ceph-mon[25742]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:52:14.089 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:52:13 smithi106 ceph-mon[25742]: Metadata not up to date on all hosts. Skipping non agent specs 2024-03-10T00:52:14.089 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:52:13 smithi106 ceph-mon[33460]: from='client.24463 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "alertmanager", "placement": "1;smithi106=a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-10T00:52:14.089 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:52:13 smithi106 ceph-mon[33460]: Saving service alertmanager spec with placement smithi106=a;count:1 2024-03-10T00:52:14.089 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:52:13 smithi106 ceph-mon[33460]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-10T00:52:14.089 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:52:13 smithi106 ceph-mon[33460]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-10T00:52:14.089 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:52:13 smithi106 ceph-mon[33460]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-10T00:52:14.089 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:52:13 smithi106 ceph-mon[33460]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:52:14.089 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:52:13 smithi106 ceph-mon[33460]: Metadata not up to date on all hosts. Skipping non agent specs 2024-03-10T00:52:14.174 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:52:13 smithi170 ceph-mon[27620]: from='client.24463 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "alertmanager", "placement": "1;smithi106=a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-10T00:52:14.175 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:52:13 smithi170 ceph-mon[27620]: Saving service alertmanager spec with placement smithi106=a;count:1 2024-03-10T00:52:14.175 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:52:13 smithi170 ceph-mon[27620]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-10T00:52:14.175 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:52:13 smithi170 ceph-mon[27620]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-10T00:52:14.175 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:52:13 smithi170 ceph-mon[27620]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-10T00:52:14.175 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:52:13 smithi170 ceph-mon[27620]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:52:14.175 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:52:13 smithi170 ceph-mon[27620]: Metadata not up to date on all hosts. Skipping non agent specs 2024-03-10T00:52:14.774 INFO:journalctl@ceph.iscsi.iscsi.a.smithi170.stdout:Mar 10 00:52:14 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-iscsi-iscsi-a[50134]: debug there is no tcmu-runner data available 2024-03-10T00:52:15.086 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:52:14 smithi170 ceph-mon[27620]: pgmap v218: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 38 KiB/s rd, 1.8 KiB/s wr, 82 op/s 2024-03-10T00:52:15.086 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:52:14 smithi170 ceph-mon[27620]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:52:15.086 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:52:14 smithi170 ceph-mon[27620]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:52:15.086 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:52:14 smithi170 ceph-mon[27620]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:52:15.086 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:52:14 smithi170 ceph-mon[27620]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-10T00:52:15.086 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:52:14 smithi170 ceph-mon[27620]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-10T00:52:15.087 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:52:14 smithi170 ceph-mon[27620]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-10T00:52:15.087 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:52:14 smithi170 ceph-mon[27620]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:52:15.309 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:52:14 smithi106 ceph-mon[25742]: pgmap v218: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 38 KiB/s rd, 1.8 KiB/s wr, 82 op/s 2024-03-10T00:52:15.309 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:52:14 smithi106 ceph-mon[25742]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:52:15.309 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:52:14 smithi106 ceph-mon[25742]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:52:15.309 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:52:14 smithi106 ceph-mon[25742]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:52:15.310 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:52:14 smithi106 ceph-mon[25742]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-10T00:52:15.310 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:52:14 smithi106 ceph-mon[25742]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-10T00:52:15.310 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:52:14 smithi106 ceph-mon[25742]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-10T00:52:15.310 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:52:14 smithi106 ceph-mon[25742]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:52:15.310 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:52:14 smithi106 ceph-mon[33460]: pgmap v218: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 38 KiB/s rd, 1.8 KiB/s wr, 82 op/s 2024-03-10T00:52:15.310 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:52:14 smithi106 ceph-mon[33460]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:52:15.310 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:52:14 smithi106 ceph-mon[33460]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:52:15.310 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:52:14 smithi106 ceph-mon[33460]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:52:15.310 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:52:14 smithi106 ceph-mon[33460]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-10T00:52:15.310 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:52:14 smithi106 ceph-mon[33460]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-10T00:52:15.311 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:52:14 smithi106 ceph-mon[33460]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-10T00:52:15.311 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:52:14 smithi106 ceph-mon[33460]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:52:15.841 INFO:teuthology.orchestra.run.smithi170.stdout:Scheduled grafana update... 2024-03-10T00:52:16.085 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:52:15 smithi106 ceph-mon[25742]: Deploying daemon prometheus.a on smithi170 2024-03-10T00:52:16.085 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:52:15 smithi106 ceph-mon[25742]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-10T00:52:16.085 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:52:15 smithi106 ceph-mon[25742]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:52:16.086 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:52:15 smithi106 ceph-mon[33460]: Deploying daemon prometheus.a on smithi170 2024-03-10T00:52:16.086 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:52:15 smithi106 ceph-mon[33460]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-10T00:52:16.086 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:52:15 smithi106 ceph-mon[33460]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:52:16.169 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:52:15 smithi170 ceph-mon[27620]: Deploying daemon prometheus.a on smithi170 2024-03-10T00:52:16.169 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:52:15 smithi170 ceph-mon[27620]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-10T00:52:16.169 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:52:15 smithi170 ceph-mon[27620]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:52:16.656 DEBUG:teuthology.orchestra.run.smithi170:grafana.a> sudo journalctl -f -n 0 -u ceph-75c37498-de77-11ee-95c7-87774f69a715@grafana.a.service 2024-03-10T00:52:16.659 INFO:tasks.cephadm:Setting up client nodes... 2024-03-10T00:52:16.659 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:831bc188fa69f143bbba69ac76119e784500e8e3 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 75c37498-de77-11ee-95c7-87774f69a715 -- ceph auth get-or-create client.0 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2024-03-10T00:52:16.903 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/75c37498-de77-11ee-95c7-87774f69a715/mon.a/config 2024-03-10T00:52:17.024 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:52:16 smithi170 ceph-mon[27620]: pgmap v219: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 3.0 KiB/s rd, 101 B/s wr, 5 op/s 2024-03-10T00:52:17.024 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:52:16 smithi170 ceph-mon[27620]: from='client.24469 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "grafana", "placement": "1;smithi170=a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-10T00:52:17.024 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:52:16 smithi170 ceph-mon[27620]: Saving service grafana spec with placement smithi170=a;count:1 2024-03-10T00:52:17.024 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:52:16 smithi170 ceph-mon[27620]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:52:17.024 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:52:16 smithi170 ceph-mon[27620]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:52:17.025 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:52:16 smithi170 ceph-mon[27620]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:52:17.025 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:52:16 smithi106 ceph-mon[33460]: pgmap v219: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 3.0 KiB/s rd, 101 B/s wr, 5 op/s 2024-03-10T00:52:17.025 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:52:16 smithi106 ceph-mon[33460]: from='client.24469 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "grafana", "placement": "1;smithi170=a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-10T00:52:17.025 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:52:16 smithi106 ceph-mon[33460]: Saving service grafana spec with placement smithi170=a;count:1 2024-03-10T00:52:17.025 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:52:16 smithi106 ceph-mon[33460]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:52:17.025 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:52:16 smithi106 ceph-mon[33460]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:52:17.025 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:52:16 smithi106 ceph-mon[33460]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:52:17.026 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:52:16 smithi106 ceph-mon[25742]: pgmap v219: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 3.0 KiB/s rd, 101 B/s wr, 5 op/s 2024-03-10T00:52:17.026 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:52:16 smithi106 ceph-mon[25742]: from='client.24469 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "grafana", "placement": "1;smithi170=a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-10T00:52:17.026 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:52:16 smithi106 ceph-mon[25742]: Saving service grafana spec with placement smithi170=a;count:1 2024-03-10T00:52:17.026 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:52:16 smithi106 ceph-mon[25742]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:52:17.026 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:52:16 smithi106 ceph-mon[25742]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:52:17.026 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:52:16 smithi106 ceph-mon[25742]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:52:19.013 INFO:teuthology.orchestra.run.smithi106.stdout:[client.0] 2024-03-10T00:52:19.014 INFO:teuthology.orchestra.run.smithi106.stdout: key = AQBDBO1l0mbEABAAEyC3PPbMotnkewhOMwBvow== 2024-03-10T00:52:19.274 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:52:18 smithi170 ceph-mon[27620]: pgmap v220: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 85 B/s wr, 4 op/s 2024-03-10T00:52:19.314 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:52:18 smithi106 ceph-mon[25742]: pgmap v220: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 85 B/s wr, 4 op/s 2024-03-10T00:52:19.314 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:52:18 smithi106 ceph-mon[33460]: pgmap v220: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 85 B/s wr, 4 op/s 2024-03-10T00:52:19.690 DEBUG:teuthology.orchestra.run.smithi106:> set -ex 2024-03-10T00:52:19.691 DEBUG:teuthology.orchestra.run.smithi106:> sudo dd of=/etc/ceph/ceph.client.0.keyring 2024-03-10T00:52:19.691 DEBUG:teuthology.orchestra.run.smithi106:> sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-03-10T00:52:19.733 DEBUG:teuthology.orchestra.run.smithi170:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:831bc188fa69f143bbba69ac76119e784500e8e3 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 75c37498-de77-11ee-95c7-87774f69a715 -- ceph auth get-or-create client.1 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2024-03-10T00:52:19.965 INFO:teuthology.orchestra.run.smithi170.stderr:Inferring config /var/lib/ceph/75c37498-de77-11ee-95c7-87774f69a715/mon.b/config 2024-03-10T00:52:20.024 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:52:19 smithi170 ceph-mon[27620]: from='client.? 172.21.15.106:0/3564906838' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-10T00:52:20.024 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:52:19 smithi170 ceph-mon[27620]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-10T00:52:20.024 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:52:19 smithi170 ceph-mon[27620]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-03-10T00:52:20.088 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:52:19 smithi106 ceph-mon[25742]: from='client.? 172.21.15.106:0/3564906838' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-10T00:52:20.088 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:52:19 smithi106 ceph-mon[25742]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-10T00:52:20.088 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:52:19 smithi106 ceph-mon[25742]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-03-10T00:52:20.089 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:52:19 smithi106 ceph-mon[33460]: from='client.? 172.21.15.106:0/3564906838' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-10T00:52:20.089 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:52:19 smithi106 ceph-mon[33460]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-10T00:52:20.089 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:52:19 smithi106 ceph-mon[33460]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-03-10T00:52:21.274 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:52:20 smithi170 ceph-mon[27620]: pgmap v221: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 2.0 KiB/s rd, 85 B/s wr, 2 op/s 2024-03-10T00:52:21.338 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:52:20 smithi106 ceph-mon[25742]: pgmap v221: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 2.0 KiB/s rd, 85 B/s wr, 2 op/s 2024-03-10T00:52:21.338 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:52:20 smithi106 ceph-mon[33460]: pgmap v221: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 2.0 KiB/s rd, 85 B/s wr, 2 op/s 2024-03-10T00:52:23.191 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:52:22 smithi170 ceph-mon[27620]: pgmap v222: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-10T00:52:23.338 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:52:22 smithi106 ceph-mon[25742]: pgmap v222: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-10T00:52:23.338 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:52:22 smithi106 ceph-mon[33460]: pgmap v222: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-10T00:52:24.604 INFO:journalctl@ceph.iscsi.iscsi.a.smithi170.stdout:Mar 10 00:52:24 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-iscsi-iscsi-a[50134]: debug there is no tcmu-runner data available 2024-03-10T00:52:24.720 INFO:teuthology.orchestra.run.smithi170.stdout:[client.1] 2024-03-10T00:52:24.720 INFO:teuthology.orchestra.run.smithi170.stdout: key = AQBIBO1lX2PVKhAAcehcI/HP3VVIN1VIW137aQ== 2024-03-10T00:52:25.274 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:52:24 smithi170 ceph-mon[27620]: pgmap v223: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-10T00:52:25.274 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:52:24 smithi170 ceph-mon[27620]: from='client.? 172.21.15.170:0/583439726' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-10T00:52:25.274 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:52:24 smithi170 ceph-mon[27620]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-10T00:52:25.274 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:52:24 smithi170 ceph-mon[27620]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-03-10T00:52:25.338 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:52:24 smithi106 ceph-mon[33460]: pgmap v223: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-10T00:52:25.338 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:52:24 smithi106 ceph-mon[33460]: from='client.? 172.21.15.170:0/583439726' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-10T00:52:25.338 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:52:24 smithi106 ceph-mon[33460]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-10T00:52:25.338 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:52:24 smithi106 ceph-mon[33460]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-03-10T00:52:25.339 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:52:24 smithi106 ceph-mon[25742]: pgmap v223: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-10T00:52:25.339 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:52:24 smithi106 ceph-mon[25742]: from='client.? 172.21.15.170:0/583439726' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-10T00:52:25.339 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:52:24 smithi106 ceph-mon[25742]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-10T00:52:25.339 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:52:24 smithi106 ceph-mon[25742]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-03-10T00:52:25.437 DEBUG:teuthology.orchestra.run.smithi170:> set -ex 2024-03-10T00:52:25.437 DEBUG:teuthology.orchestra.run.smithi170:> sudo dd of=/etc/ceph/ceph.client.1.keyring 2024-03-10T00:52:25.437 DEBUG:teuthology.orchestra.run.smithi170:> sudo chmod 0644 /etc/ceph/ceph.client.1.keyring 2024-03-10T00:52:25.482 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-03-10T00:52:25.482 INFO:tasks.cephadm.ceph_manager.ceph:waiting for mgr available 2024-03-10T00:52:25.483 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:831bc188fa69f143bbba69ac76119e784500e8e3 shell --fsid 75c37498-de77-11ee-95c7-87774f69a715 -- ceph mgr dump --format=json 2024-03-10T00:52:25.732 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/75c37498-de77-11ee-95c7-87774f69a715/mon.a/config 2024-03-10T00:52:26.117 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:52:25 smithi106 ceph-mon[33460]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-10T00:52:26.117 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:52:25 smithi106 ceph-mon[25742]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-10T00:52:26.188 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:52:25 smithi170 ceph-mon[27620]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-10T00:52:27.124 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:52:26 smithi106 ceph-mon[33460]: pgmap v224: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-10T00:52:27.124 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:52:26 smithi106 ceph-mon[25742]: pgmap v224: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-10T00:52:27.166 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:52:26 smithi170 ceph-mon[27620]: pgmap v224: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-10T00:52:27.612 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-03-10T00:52:27.974 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:52:27 smithi170 ceph-mon[27620]: from='client.? 172.21.15.106:0/477659986' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-03-10T00:52:28.235 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:52:27 smithi106 ceph-mon[25742]: from='client.? 172.21.15.106:0/477659986' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-03-10T00:52:28.236 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:52:27 smithi106 ceph-mon[33460]: from='client.? 172.21.15.106:0/477659986' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-03-10T00:52:28.262 INFO:teuthology.orchestra.run.smithi106.stdout:{"epoch":16,"flags":0,"active_gid":14152,"active_name":"y","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6800","nonce":3667641346},{"type":"v1","addr":"172.21.15.106:6801","nonce":3667641346}]},"active_addr":"172.21.15.106:6801/3667641346","active_change":"2024-03-10T00:47:00.733086+0000","active_mgr_features":4540701547738038271,"available":true,"standbys":[{"gid":24115,"name":"x","mgr_features":4540701547738038271,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.0.0","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["cephadm","dashboard","iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.0.0","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{"dashboard":"https://172.21.15.106:8443/"},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"squid":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":3,"active_clients":[{"name":"cephadm","addrvec":[{"type":"v2","addr":"172.21.15.106:0","nonce":113029157}]},{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.106:0","nonce":3294075298}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.106:0","nonce":1676011394}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.106:0","nonce":3380427339}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.106:0","nonce":2534289678}]}]} 2024-03-10T00:52:28.266 INFO:tasks.cephadm.ceph_manager.ceph:mgr available! 2024-03-10T00:52:28.266 INFO:tasks.cephadm.ceph_manager.ceph:waiting for all up 2024-03-10T00:52:28.266 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:831bc188fa69f143bbba69ac76119e784500e8e3 shell --fsid 75c37498-de77-11ee-95c7-87774f69a715 -- ceph osd dump --format=json 2024-03-10T00:52:28.274 INFO:journalctl@ceph.prometheus.a.smithi170.stdout:Mar 10 00:52:28 smithi170 systemd[1]: Starting Ceph prometheus.a for 75c37498-de77-11ee-95c7-87774f69a715... 2024-03-10T00:52:28.508 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/75c37498-de77-11ee-95c7-87774f69a715/mon.a/config 2024-03-10T00:52:28.892 INFO:journalctl@ceph.prometheus.a.smithi170.stdout:Mar 10 00:52:28 smithi170 podman[52560]: 2024-03-10 00:52:28.640041956 +0000 UTC m=+0.449050032 image pull a07b618ecd1dce142bce4c52f0e80982eaf1f14265a2415c2d35978ccaa0a464 quay.io/prometheus/prometheus:v2.43.0 2024-03-10T00:52:28.892 INFO:journalctl@ceph.prometheus.a.smithi170.stdout:Mar 10 00:52:28 smithi170 podman[52560]: 2024-03-10 00:52:28.768290372 +0000 UTC m=+0.577298492 container create 13aadf83806a9cd335ab57b8cd34ded81c4876c5a3cdd8bf6ce9cb310c1b4de1 (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-75c37498-de77-11ee-95c7-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-10T00:52:29.274 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:52:28 smithi170 ceph-mon[27620]: pgmap v225: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-10T00:52:29.275 INFO:journalctl@ceph.prometheus.a.smithi170.stdout:Mar 10 00:52:29 smithi170 podman[52560]: 2024-03-10 00:52:29.037424559 +0000 UTC m=+0.846432636 container init 13aadf83806a9cd335ab57b8cd34ded81c4876c5a3cdd8bf6ce9cb310c1b4de1 (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-75c37498-de77-11ee-95c7-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-10T00:52:29.275 INFO:journalctl@ceph.prometheus.a.smithi170.stdout:Mar 10 00:52:29 smithi170 podman[52560]: 2024-03-10 00:52:29.042842968 +0000 UTC m=+0.851851057 container start 13aadf83806a9cd335ab57b8cd34ded81c4876c5a3cdd8bf6ce9cb310c1b4de1 (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-75c37498-de77-11ee-95c7-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-10T00:52:29.275 INFO:journalctl@ceph.prometheus.a.smithi170.stdout:Mar 10 00:52:29 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-prometheus-a[52583]: ts=2024-03-10T00:52:29.067Z caller=main.go:564 level=info msg="Starting Prometheus Server" mode=server version="(version=2.43.0, branch=HEAD, revision=edfc3bcd025dd6fe296c167a14a216cab1e552ee)" 2024-03-10T00:52:29.275 INFO:journalctl@ceph.prometheus.a.smithi170.stdout:Mar 10 00:52:29 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-prometheus-a[52583]: ts=2024-03-10T00:52:29.067Z caller=main.go:569 level=info build_context="(go=go1.19.7, platform=linux/amd64, user=root@8a0ee342e522, date=20230321-12:56:07, tags=netgo,builtinassets)" 2024-03-10T00:52:29.275 INFO:journalctl@ceph.prometheus.a.smithi170.stdout:Mar 10 00:52:29 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-prometheus-a[52583]: ts=2024-03-10T00:52:29.067Z caller=main.go:570 level=info host_details="(Linux 5.14.0-427.el9.x86_64 #1 SMP PREEMPT_DYNAMIC Fri Feb 23 04:45:07 UTC 2024 x86_64 smithi170 (none))" 2024-03-10T00:52:29.275 INFO:journalctl@ceph.prometheus.a.smithi170.stdout:Mar 10 00:52:29 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-prometheus-a[52583]: ts=2024-03-10T00:52:29.067Z caller=main.go:571 level=info fd_limits="(soft=1048576, hard=1048576)" 2024-03-10T00:52:29.275 INFO:journalctl@ceph.prometheus.a.smithi170.stdout:Mar 10 00:52:29 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-prometheus-a[52583]: ts=2024-03-10T00:52:29.067Z caller=main.go:572 level=info vm_limits="(soft=unlimited, hard=unlimited)" 2024-03-10T00:52:29.275 INFO:journalctl@ceph.prometheus.a.smithi170.stdout:Mar 10 00:52:29 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-prometheus-a[52583]: ts=2024-03-10T00:52:29.068Z caller=web.go:561 level=info component=web msg="Start listening for connections" address=:9095 2024-03-10T00:52:29.275 INFO:journalctl@ceph.prometheus.a.smithi170.stdout:Mar 10 00:52:29 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-prometheus-a[52583]: ts=2024-03-10T00:52:29.069Z caller=main.go:1005 level=info msg="Starting TSDB ..." 2024-03-10T00:52:29.275 INFO:journalctl@ceph.prometheus.a.smithi170.stdout:Mar 10 00:52:29 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-prometheus-a[52583]: ts=2024-03-10T00:52:29.070Z caller=tls_config.go:232 level=info component=web msg="Listening on" address=[::]:9095 2024-03-10T00:52:29.275 INFO:journalctl@ceph.prometheus.a.smithi170.stdout:Mar 10 00:52:29 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-prometheus-a[52583]: ts=2024-03-10T00:52:29.070Z caller=tls_config.go:235 level=info component=web msg="TLS is disabled." http2=false address=[::]:9095 2024-03-10T00:52:29.276 INFO:journalctl@ceph.prometheus.a.smithi170.stdout:Mar 10 00:52:29 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-prometheus-a[52583]: ts=2024-03-10T00:52:29.073Z caller=head.go:587 level=info component=tsdb msg="Replaying on-disk memory mappable chunks if any" 2024-03-10T00:52:29.276 INFO:journalctl@ceph.prometheus.a.smithi170.stdout:Mar 10 00:52:29 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-prometheus-a[52583]: ts=2024-03-10T00:52:29.073Z caller=head.go:658 level=info component=tsdb msg="On-disk memory mappable chunks replay completed" duration=1.728µs 2024-03-10T00:52:29.276 INFO:journalctl@ceph.prometheus.a.smithi170.stdout:Mar 10 00:52:29 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-prometheus-a[52583]: ts=2024-03-10T00:52:29.073Z caller=head.go:664 level=info component=tsdb msg="Replaying WAL, this may take a while" 2024-03-10T00:52:29.276 INFO:journalctl@ceph.prometheus.a.smithi170.stdout:Mar 10 00:52:29 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-prometheus-a[52583]: ts=2024-03-10T00:52:29.073Z caller=head.go:735 level=info component=tsdb msg="WAL segment loaded" segment=0 maxSegment=0 2024-03-10T00:52:29.276 INFO:journalctl@ceph.prometheus.a.smithi170.stdout:Mar 10 00:52:29 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-prometheus-a[52583]: ts=2024-03-10T00:52:29.073Z caller=head.go:772 level=info component=tsdb msg="WAL replay completed" checkpoint_replay_duration=18.348µs wal_replay_duration=191.877µs wbl_replay_duration=137ns total_replay_duration=228.156µs 2024-03-10T00:52:29.276 INFO:journalctl@ceph.prometheus.a.smithi170.stdout:Mar 10 00:52:29 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-prometheus-a[52583]: ts=2024-03-10T00:52:29.074Z caller=main.go:1026 level=info fs_type=XFS_SUPER_MAGIC 2024-03-10T00:52:29.276 INFO:journalctl@ceph.prometheus.a.smithi170.stdout:Mar 10 00:52:29 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-prometheus-a[52583]: ts=2024-03-10T00:52:29.074Z caller=main.go:1029 level=info msg="TSDB started" 2024-03-10T00:52:29.276 INFO:journalctl@ceph.prometheus.a.smithi170.stdout:Mar 10 00:52:29 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-prometheus-a[52583]: ts=2024-03-10T00:52:29.074Z caller=main.go:1209 level=info msg="Loading configuration file" filename=/etc/prometheus/prometheus.yml 2024-03-10T00:52:29.276 INFO:journalctl@ceph.prometheus.a.smithi170.stdout:Mar 10 00:52:29 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-prometheus-a[52583]: ts=2024-03-10T00:52:29.095Z caller=main.go:1246 level=info msg="Completed loading of configuration file" filename=/etc/prometheus/prometheus.yml totalDuration=20.898622ms db_storage=835ns remote_storage=1.24µs web_handler=332ns query_engine=557ns scrape=410.337µs scrape_sd=75.764µs notify=1.02µs notify_sd=1.94µs rules=20.022118ms tracing=7.013µs 2024-03-10T00:52:29.276 INFO:journalctl@ceph.prometheus.a.smithi170.stdout:Mar 10 00:52:29 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-prometheus-a[52583]: ts=2024-03-10T00:52:29.095Z caller=main.go:990 level=info msg="Server is ready to receive web requests." 2024-03-10T00:52:29.276 INFO:journalctl@ceph.prometheus.a.smithi170.stdout:Mar 10 00:52:29 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-prometheus-a[52583]: ts=2024-03-10T00:52:29.095Z caller=manager.go:974 level=info component="rule manager" msg="Starting rule manager..." 2024-03-10T00:52:29.276 INFO:journalctl@ceph.prometheus.a.smithi170.stdout:Mar 10 00:52:29 smithi170 bash[52560]: 13aadf83806a9cd335ab57b8cd34ded81c4876c5a3cdd8bf6ce9cb310c1b4de1 2024-03-10T00:52:29.276 INFO:journalctl@ceph.prometheus.a.smithi170.stdout:Mar 10 00:52:29 smithi170 systemd[1]: Started Ceph prometheus.a for 75c37498-de77-11ee-95c7-87774f69a715. 2024-03-10T00:52:29.338 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:52:28 smithi106 ceph-mon[25742]: pgmap v225: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-10T00:52:29.338 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:52:28 smithi106 ceph-mon[33460]: pgmap v225: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-10T00:52:30.241 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:52:30 smithi170 ceph-mon[27620]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:52:30.241 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:52:30 smithi170 ceph-mon[27620]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:52:30.241 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:52:30 smithi170 ceph-mon[27620]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:52:30.241 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:52:30 smithi170 ceph-mon[27620]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "mgr module enable", "module": "prometheus"}]: dispatch 2024-03-10T00:52:30.241 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:52:30 smithi170 ceph-mon[27620]: pgmap v226: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-10T00:52:30.358 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-03-10T00:52:30.358 INFO:teuthology.orchestra.run.smithi106.stdout:{"epoch":68,"fsid":"75c37498-de77-11ee-95c7-87774f69a715","created":"2024-03-10T00:46:03.418276+0000","modified":"2024-03-10T00:52:05.370170+0000","last_up_change":"2024-03-10T00:51:37.801567+0000","last_in_change":"2024-03-10T00:51:18.974575+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":18,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":6,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-10T00:49:25.071132+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":7.8899998664855957,"score_stable":7.8899998664855957,"optimal_score":0.37999999523162842,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"datapool","create_time":"2024-03-10T00:51:51.848096+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":3,"pg_placement_num":3,"pg_placement_num_target":3,"pg_num_target":3,"pg_num_pending":3,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"64","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":64,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.6500000953674316,"score_stable":2.6500000953674316,"optimal_score":0.87999999523162842,"raw_score_acting":2.3299999237060547,"raw_score_stable":2.3299999237060547,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":".rgw.root","create_time":"2024-03-10T00:51:52.085395+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"61","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"default.rgw.log","create_time":"2024-03-10T00:51:53.428382+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"63","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"default.rgw.control","create_time":"2024-03-10T00:51:55.365722+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"65","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"default.rgw.meta","create_time":"2024-03-10T00:51:57.427600+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"67","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"90836761-e042-415d-b610-3a8383a65d57","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6802","nonce":3397660587},{"type":"v1","addr":"172.21.15.106:6803","nonce":3397660587}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6804","nonce":3397660587},{"type":"v1","addr":"172.21.15.106:6805","nonce":3397660587}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6808","nonce":3397660587},{"type":"v1","addr":"172.21.15.106:6809","nonce":3397660587}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6806","nonce":3397660587},{"type":"v1","addr":"172.21.15.106:6807","nonce":3397660587}]},"public_addr":"172.21.15.106:6803/3397660587","cluster_addr":"172.21.15.106:6805/3397660587","heartbeat_back_addr":"172.21.15.106:6809/3397660587","heartbeat_front_addr":"172.21.15.106:6807/3397660587","state":["exists","up"]},{"osd":1,"uuid":"207e1815-38e1-498c-b0cd-602a70cdffa1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6810","nonce":270281423},{"type":"v1","addr":"172.21.15.106:6811","nonce":270281423}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6812","nonce":270281423},{"type":"v1","addr":"172.21.15.106:6813","nonce":270281423}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6816","nonce":270281423},{"type":"v1","addr":"172.21.15.106:6817","nonce":270281423}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6814","nonce":270281423},{"type":"v1","addr":"172.21.15.106:6815","nonce":270281423}]},"public_addr":"172.21.15.106:6811/270281423","cluster_addr":"172.21.15.106:6813/270281423","heartbeat_back_addr":"172.21.15.106:6817/270281423","heartbeat_front_addr":"172.21.15.106:6815/270281423","state":["exists","up"]},{"osd":2,"uuid":"b8113c87-8517-4b71-b6b4-ad7f2fce167b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6818","nonce":3393056025},{"type":"v1","addr":"172.21.15.106:6819","nonce":3393056025}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6820","nonce":3393056025},{"type":"v1","addr":"172.21.15.106:6821","nonce":3393056025}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6824","nonce":3393056025},{"type":"v1","addr":"172.21.15.106:6825","nonce":3393056025}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6822","nonce":3393056025},{"type":"v1","addr":"172.21.15.106:6823","nonce":3393056025}]},"public_addr":"172.21.15.106:6819/3393056025","cluster_addr":"172.21.15.106:6821/3393056025","heartbeat_back_addr":"172.21.15.106:6825/3393056025","heartbeat_front_addr":"172.21.15.106:6823/3393056025","state":["exists","up"]},{"osd":3,"uuid":"b548c1c5-2658-47f6-89a3-f52f3e8c0758","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6826","nonce":2289173680},{"type":"v1","addr":"172.21.15.106:6827","nonce":2289173680}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6828","nonce":2289173680},{"type":"v1","addr":"172.21.15.106:6829","nonce":2289173680}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6832","nonce":2289173680},{"type":"v1","addr":"172.21.15.106:6833","nonce":2289173680}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6830","nonce":2289173680},{"type":"v1","addr":"172.21.15.106:6831","nonce":2289173680}]},"public_addr":"172.21.15.106:6827/2289173680","cluster_addr":"172.21.15.106:6829/2289173680","heartbeat_back_addr":"172.21.15.106:6833/2289173680","heartbeat_front_addr":"172.21.15.106:6831/2289173680","state":["exists","up"]},{"osd":4,"uuid":"4fcacd32-e9ca-4aa7-b99e-bac33fc58754","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":30,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6800","nonce":335809953},{"type":"v1","addr":"172.21.15.170:6801","nonce":335809953}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6802","nonce":335809953},{"type":"v1","addr":"172.21.15.170:6803","nonce":335809953}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6806","nonce":335809953},{"type":"v1","addr":"172.21.15.170:6807","nonce":335809953}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6804","nonce":335809953},{"type":"v1","addr":"172.21.15.170:6805","nonce":335809953}]},"public_addr":"172.21.15.170:6801/335809953","cluster_addr":"172.21.15.170:6803/335809953","heartbeat_back_addr":"172.21.15.170:6807/335809953","heartbeat_front_addr":"172.21.15.170:6805/335809953","state":["exists","up"]},{"osd":5,"uuid":"cf14b75a-d753-4279-9b67-593fed3c5785","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6808","nonce":47584024},{"type":"v1","addr":"172.21.15.170:6809","nonce":47584024}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6810","nonce":47584024},{"type":"v1","addr":"172.21.15.170:6811","nonce":47584024}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6814","nonce":47584024},{"type":"v1","addr":"172.21.15.170:6815","nonce":47584024}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6812","nonce":47584024},{"type":"v1","addr":"172.21.15.170:6813","nonce":47584024}]},"public_addr":"172.21.15.170:6809/47584024","cluster_addr":"172.21.15.170:6811/47584024","heartbeat_back_addr":"172.21.15.170:6815/47584024","heartbeat_front_addr":"172.21.15.170:6813/47584024","state":["exists","up"]},{"osd":6,"uuid":"46e00786-a5d2-48fe-b750-89beb0d63194","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":63,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6816","nonce":814825309},{"type":"v1","addr":"172.21.15.170:6817","nonce":814825309}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6818","nonce":814825309},{"type":"v1","addr":"172.21.15.170:6819","nonce":814825309}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6822","nonce":814825309},{"type":"v1","addr":"172.21.15.170:6823","nonce":814825309}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6820","nonce":814825309},{"type":"v1","addr":"172.21.15.170:6821","nonce":814825309}]},"public_addr":"172.21.15.170:6817/814825309","cluster_addr":"172.21.15.170:6819/814825309","heartbeat_back_addr":"172.21.15.170:6823/814825309","heartbeat_front_addr":"172.21.15.170:6821/814825309","state":["exists","up"]},{"osd":7,"uuid":"56567866-297b-457f-922a-7a53a1f3ef21","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":54,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6824","nonce":472398409},{"type":"v1","addr":"172.21.15.170:6825","nonce":472398409}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6826","nonce":472398409},{"type":"v1","addr":"172.21.15.170:6827","nonce":472398409}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6830","nonce":472398409},{"type":"v1","addr":"172.21.15.170:6831","nonce":472398409}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6828","nonce":472398409},{"type":"v1","addr":"172.21.15.170:6829","nonce":472398409}]},"public_addr":"172.21.15.170:6825/472398409","cluster_addr":"172.21.15.170:6827/472398409","heartbeat_back_addr":"172.21.15.170:6831/472398409","heartbeat_front_addr":"172.21.15.170:6829/472398409","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-10T00:48:30.232907+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-10T00:48:56.221202+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-10T00:49:22.529778+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-10T00:49:52.090754+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-10T00:50:16.772217+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-10T00:50:42.035375+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-10T00:51:09.412164+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-10T00:51:35.931598+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.106:0/3982476751":"2024-03-11T00:47:00.732937+0000","172.21.15.106:6801/2449347734":"2024-03-11T00:47:00.732937+0000","172.21.15.106:6800/2449347734":"2024-03-11T00:47:00.732937+0000","172.21.15.106:0/3555678354":"2024-03-11T00:47:00.732937+0000","172.21.15.106:6801/1073099015":"2024-03-11T00:46:33.968987+0000","172.21.15.106:0/4150429375":"2024-03-11T00:46:33.968987+0000","172.21.15.106:6800/1073099015":"2024-03-11T00:46:33.968987+0000","172.21.15.106:0/232887326":"2024-03-11T00:46:33.968987+0000","172.21.15.106:0/4228752537":"2024-03-11T00:46:33.968987+0000","172.21.15.106:0/3721367154":"2024-03-11T00:47:00.732937+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-10T00:52:30.524 INFO:journalctl@ceph.mgr.x.smithi170.stdout:Mar 10 00:52:30 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-mgr-x[30157]: ignoring --setuser ceph since I am not root 2024-03-10T00:52:30.524 INFO:journalctl@ceph.mgr.x.smithi170.stdout:Mar 10 00:52:30 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-mgr-x[30157]: ignoring --setgroup ceph since I am not root 2024-03-10T00:52:30.524 INFO:journalctl@ceph.mgr.x.smithi170.stdout:Mar 10 00:52:30 smithi170 ceph-mgr[30180]: -- 172.21.15.170:0/2104187081 <== mon.1 v2:172.21.15.170:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55e4f508a5a0 con 0x55e4f505f000 2024-03-10T00:52:30.588 INFO:journalctl@ceph.mgr.y.smithi106.stdout:Mar 10 00:52:30 smithi106 ceph-75c37498-de77-11ee-95c7-87774f69a715-mgr-y[25997]: ignoring --setuser ceph since I am not root 2024-03-10T00:52:30.588 INFO:journalctl@ceph.mgr.y.smithi106.stdout:Mar 10 00:52:30 smithi106 ceph-75c37498-de77-11ee-95c7-87774f69a715-mgr-y[25997]: ignoring --setgroup ceph since I am not root 2024-03-10T00:52:30.588 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:52:30 smithi106 ceph-mon[33460]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:52:30.589 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:52:30 smithi106 ceph-mon[33460]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:52:30.589 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:52:30 smithi106 ceph-mon[33460]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:52:30.589 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:52:30 smithi106 ceph-mon[33460]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "mgr module enable", "module": "prometheus"}]: dispatch 2024-03-10T00:52:30.589 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:52:30 smithi106 ceph-mon[33460]: pgmap v226: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-10T00:52:30.589 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:52:30 smithi106 ceph-mon[25742]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:52:30.589 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:52:30 smithi106 ceph-mon[25742]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:52:30.589 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:52:30 smithi106 ceph-mon[25742]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' 2024-03-10T00:52:30.589 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:52:30 smithi106 ceph-mon[25742]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd=[{"prefix": "mgr module enable", "module": "prometheus"}]: dispatch 2024-03-10T00:52:30.589 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:52:30 smithi106 ceph-mon[25742]: pgmap v226: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-10T00:52:30.873 INFO:journalctl@ceph.mgr.y.smithi106.stdout:Mar 10 00:52:30 smithi106 ceph-75c37498-de77-11ee-95c7-87774f69a715-mgr-y[25997]: 2024-03-10T00:52:30.780+0000 7fd00781c200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-10T00:52:31.024 INFO:journalctl@ceph.mgr.x.smithi170.stdout:Mar 10 00:52:30 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-mgr-x[30157]: 2024-03-10T00:52:30.823+0000 7f1a88cc8200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-10T00:52:31.024 INFO:journalctl@ceph.mgr.x.smithi170.stdout:Mar 10 00:52:30 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-mgr-x[30157]: 2024-03-10T00:52:30.919+0000 7f1a88cc8200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-10T00:52:31.094 INFO:tasks.cephadm.ceph_manager.ceph:all up! 2024-03-10T00:52:31.095 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:831bc188fa69f143bbba69ac76119e784500e8e3 shell --fsid 75c37498-de77-11ee-95c7-87774f69a715 -- ceph osd dump --format=json 2024-03-10T00:52:31.205 INFO:journalctl@ceph.mgr.y.smithi106.stdout:Mar 10 00:52:30 smithi106 ceph-75c37498-de77-11ee-95c7-87774f69a715-mgr-y[25997]: 2024-03-10T00:52:30.875+0000 7fd00781c200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-10T00:52:31.349 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/75c37498-de77-11ee-95c7-87774f69a715/mon.a/config 2024-03-10T00:52:31.466 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:52:31 smithi106 ceph-mon[25742]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd='[{"prefix": "mgr module enable", "module": "prometheus"}]': finished 2024-03-10T00:52:31.466 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:52:31 smithi106 ceph-mon[25742]: mgrmap e17: y(active, since 5m), standbys: x 2024-03-10T00:52:31.466 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:52:31 smithi106 ceph-mon[25742]: from='client.? 172.21.15.106:0/1700318380' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-10T00:52:31.467 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:52:31 smithi106 ceph-mon[33460]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd='[{"prefix": "mgr module enable", "module": "prometheus"}]': finished 2024-03-10T00:52:31.467 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:52:31 smithi106 ceph-mon[33460]: mgrmap e17: y(active, since 5m), standbys: x 2024-03-10T00:52:31.467 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:52:31 smithi106 ceph-mon[33460]: from='client.? 172.21.15.106:0/1700318380' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-10T00:52:31.524 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:52:31 smithi170 ceph-mon[27620]: from='mgr.14152 172.21.15.106:0/1209273813' entity='mgr.y' cmd='[{"prefix": "mgr module enable", "module": "prometheus"}]': finished 2024-03-10T00:52:31.524 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:52:31 smithi170 ceph-mon[27620]: mgrmap e17: y(active, since 5m), standbys: x 2024-03-10T00:52:31.524 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:52:31 smithi170 ceph-mon[27620]: from='client.? 172.21.15.106:0/1700318380' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-10T00:52:31.744 INFO:journalctl@ceph.mgr.y.smithi106.stdout:Mar 10 00:52:31 smithi106 ceph-75c37498-de77-11ee-95c7-87774f69a715-mgr-y[25997]: 2024-03-10T00:52:31.548+0000 7fd00781c200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-10T00:52:31.744 INFO:journalctl@ceph.mgr.y.smithi106.stdout:Mar 10 00:52:31 smithi106 ceph-75c37498-de77-11ee-95c7-87774f69a715-mgr-y[25997]: 2024-03-10T00:52:31.646+0000 7fd00781c200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-10T00:52:31.804 INFO:journalctl@ceph.mgr.x.smithi170.stdout:Mar 10 00:52:31 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-mgr-x[30157]: 2024-03-10T00:52:31.568+0000 7f1a88cc8200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-10T00:52:31.804 INFO:journalctl@ceph.mgr.x.smithi170.stdout:Mar 10 00:52:31 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-mgr-x[30157]: 2024-03-10T00:52:31.661+0000 7f1a88cc8200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-10T00:52:32.088 INFO:journalctl@ceph.mgr.y.smithi106.stdout:Mar 10 00:52:31 smithi106 ceph-75c37498-de77-11ee-95c7-87774f69a715-mgr-y[25997]: 2024-03-10T00:52:31.792+0000 7fd00781c200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-10T00:52:32.088 INFO:journalctl@ceph.mgr.y.smithi106.stdout:Mar 10 00:52:31 smithi106 ceph-75c37498-de77-11ee-95c7-87774f69a715-mgr-y[25997]: 2024-03-10T00:52:31.888+0000 7fd00781c200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-10T00:52:32.088 INFO:journalctl@ceph.mgr.y.smithi106.stdout:Mar 10 00:52:31 smithi106 ceph-75c37498-de77-11ee-95c7-87774f69a715-mgr-y[25997]: 2024-03-10T00:52:31.971+0000 7fd00781c200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-10T00:52:32.274 INFO:journalctl@ceph.mgr.x.smithi170.stdout:Mar 10 00:52:31 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-mgr-x[30157]: 2024-03-10T00:52:31.806+0000 7f1a88cc8200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-10T00:52:32.274 INFO:journalctl@ceph.mgr.x.smithi170.stdout:Mar 10 00:52:31 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-mgr-x[30157]: 2024-03-10T00:52:31.901+0000 7f1a88cc8200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-10T00:52:32.274 INFO:journalctl@ceph.mgr.x.smithi170.stdout:Mar 10 00:52:31 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-mgr-x[30157]: 2024-03-10T00:52:31.990+0000 7f1a88cc8200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-10T00:52:32.536 INFO:journalctl@ceph.mgr.y.smithi106.stdout:Mar 10 00:52:32 smithi106 ceph-75c37498-de77-11ee-95c7-87774f69a715-mgr-y[25997]: 2024-03-10T00:52:32.243+0000 7fd00781c200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-10T00:52:32.536 INFO:journalctl@ceph.mgr.y.smithi106.stdout:Mar 10 00:52:32 smithi106 ceph-75c37498-de77-11ee-95c7-87774f69a715-mgr-y[25997]: 2024-03-10T00:52:32.352+0000 7fd00781c200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-10T00:52:32.536 INFO:journalctl@ceph.mgr.y.smithi106.stdout:Mar 10 00:52:32 smithi106 ceph-75c37498-de77-11ee-95c7-87774f69a715-mgr-y[25997]: 2024-03-10T00:52:32.440+0000 7fd00781c200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-10T00:52:32.536 INFO:journalctl@ceph.mgr.y.smithi106.stdout:Mar 10 00:52:32 smithi106 ceph-75c37498-de77-11ee-95c7-87774f69a715-mgr-y[25997]: 2024-03-10T00:52:32.538+0000 7fd00781c200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-10T00:52:32.578 INFO:journalctl@ceph.mgr.x.smithi170.stdout:Mar 10 00:52:32 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-mgr-x[30157]: 2024-03-10T00:52:32.279+0000 7f1a88cc8200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-10T00:52:32.578 INFO:journalctl@ceph.mgr.x.smithi170.stdout:Mar 10 00:52:32 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-mgr-x[30157]: 2024-03-10T00:52:32.389+0000 7f1a88cc8200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-10T00:52:32.578 INFO:journalctl@ceph.mgr.x.smithi170.stdout:Mar 10 00:52:32 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-mgr-x[30157]: 2024-03-10T00:52:32.482+0000 7f1a88cc8200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-10T00:52:32.838 INFO:journalctl@ceph.mgr.y.smithi106.stdout:Mar 10 00:52:32 smithi106 ceph-75c37498-de77-11ee-95c7-87774f69a715-mgr-y[25997]: 2024-03-10T00:52:32.720+0000 7fd00781c200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-10T00:52:33.024 INFO:journalctl@ceph.mgr.x.smithi170.stdout:Mar 10 00:52:32 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-mgr-x[30157]: 2024-03-10T00:52:32.580+0000 7f1a88cc8200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-10T00:52:33.024 INFO:journalctl@ceph.mgr.x.smithi170.stdout:Mar 10 00:52:32 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-mgr-x[30157]: 2024-03-10T00:52:32.761+0000 7f1a88cc8200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-10T00:52:33.347 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-03-10T00:52:33.348 INFO:teuthology.orchestra.run.smithi106.stdout:{"epoch":68,"fsid":"75c37498-de77-11ee-95c7-87774f69a715","created":"2024-03-10T00:46:03.418276+0000","modified":"2024-03-10T00:52:05.370170+0000","last_up_change":"2024-03-10T00:51:37.801567+0000","last_in_change":"2024-03-10T00:51:18.974575+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":18,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":6,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-10T00:49:25.071132+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":7.8899998664855957,"score_stable":7.8899998664855957,"optimal_score":0.37999999523162842,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"datapool","create_time":"2024-03-10T00:51:51.848096+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":3,"pg_placement_num":3,"pg_placement_num_target":3,"pg_num_target":3,"pg_num_pending":3,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"64","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":64,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.6500000953674316,"score_stable":2.6500000953674316,"optimal_score":0.87999999523162842,"raw_score_acting":2.3299999237060547,"raw_score_stable":2.3299999237060547,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":".rgw.root","create_time":"2024-03-10T00:51:52.085395+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"61","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"default.rgw.log","create_time":"2024-03-10T00:51:53.428382+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"63","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"default.rgw.control","create_time":"2024-03-10T00:51:55.365722+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"65","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"default.rgw.meta","create_time":"2024-03-10T00:51:57.427600+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"67","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"90836761-e042-415d-b610-3a8383a65d57","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6802","nonce":3397660587},{"type":"v1","addr":"172.21.15.106:6803","nonce":3397660587}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6804","nonce":3397660587},{"type":"v1","addr":"172.21.15.106:6805","nonce":3397660587}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6808","nonce":3397660587},{"type":"v1","addr":"172.21.15.106:6809","nonce":3397660587}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6806","nonce":3397660587},{"type":"v1","addr":"172.21.15.106:6807","nonce":3397660587}]},"public_addr":"172.21.15.106:6803/3397660587","cluster_addr":"172.21.15.106:6805/3397660587","heartbeat_back_addr":"172.21.15.106:6809/3397660587","heartbeat_front_addr":"172.21.15.106:6807/3397660587","state":["exists","up"]},{"osd":1,"uuid":"207e1815-38e1-498c-b0cd-602a70cdffa1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6810","nonce":270281423},{"type":"v1","addr":"172.21.15.106:6811","nonce":270281423}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6812","nonce":270281423},{"type":"v1","addr":"172.21.15.106:6813","nonce":270281423}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6816","nonce":270281423},{"type":"v1","addr":"172.21.15.106:6817","nonce":270281423}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6814","nonce":270281423},{"type":"v1","addr":"172.21.15.106:6815","nonce":270281423}]},"public_addr":"172.21.15.106:6811/270281423","cluster_addr":"172.21.15.106:6813/270281423","heartbeat_back_addr":"172.21.15.106:6817/270281423","heartbeat_front_addr":"172.21.15.106:6815/270281423","state":["exists","up"]},{"osd":2,"uuid":"b8113c87-8517-4b71-b6b4-ad7f2fce167b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6818","nonce":3393056025},{"type":"v1","addr":"172.21.15.106:6819","nonce":3393056025}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6820","nonce":3393056025},{"type":"v1","addr":"172.21.15.106:6821","nonce":3393056025}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6824","nonce":3393056025},{"type":"v1","addr":"172.21.15.106:6825","nonce":3393056025}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6822","nonce":3393056025},{"type":"v1","addr":"172.21.15.106:6823","nonce":3393056025}]},"public_addr":"172.21.15.106:6819/3393056025","cluster_addr":"172.21.15.106:6821/3393056025","heartbeat_back_addr":"172.21.15.106:6825/3393056025","heartbeat_front_addr":"172.21.15.106:6823/3393056025","state":["exists","up"]},{"osd":3,"uuid":"b548c1c5-2658-47f6-89a3-f52f3e8c0758","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6826","nonce":2289173680},{"type":"v1","addr":"172.21.15.106:6827","nonce":2289173680}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6828","nonce":2289173680},{"type":"v1","addr":"172.21.15.106:6829","nonce":2289173680}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6832","nonce":2289173680},{"type":"v1","addr":"172.21.15.106:6833","nonce":2289173680}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6830","nonce":2289173680},{"type":"v1","addr":"172.21.15.106:6831","nonce":2289173680}]},"public_addr":"172.21.15.106:6827/2289173680","cluster_addr":"172.21.15.106:6829/2289173680","heartbeat_back_addr":"172.21.15.106:6833/2289173680","heartbeat_front_addr":"172.21.15.106:6831/2289173680","state":["exists","up"]},{"osd":4,"uuid":"4fcacd32-e9ca-4aa7-b99e-bac33fc58754","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":30,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6800","nonce":335809953},{"type":"v1","addr":"172.21.15.170:6801","nonce":335809953}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6802","nonce":335809953},{"type":"v1","addr":"172.21.15.170:6803","nonce":335809953}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6806","nonce":335809953},{"type":"v1","addr":"172.21.15.170:6807","nonce":335809953}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6804","nonce":335809953},{"type":"v1","addr":"172.21.15.170:6805","nonce":335809953}]},"public_addr":"172.21.15.170:6801/335809953","cluster_addr":"172.21.15.170:6803/335809953","heartbeat_back_addr":"172.21.15.170:6807/335809953","heartbeat_front_addr":"172.21.15.170:6805/335809953","state":["exists","up"]},{"osd":5,"uuid":"cf14b75a-d753-4279-9b67-593fed3c5785","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6808","nonce":47584024},{"type":"v1","addr":"172.21.15.170:6809","nonce":47584024}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6810","nonce":47584024},{"type":"v1","addr":"172.21.15.170:6811","nonce":47584024}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6814","nonce":47584024},{"type":"v1","addr":"172.21.15.170:6815","nonce":47584024}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6812","nonce":47584024},{"type":"v1","addr":"172.21.15.170:6813","nonce":47584024}]},"public_addr":"172.21.15.170:6809/47584024","cluster_addr":"172.21.15.170:6811/47584024","heartbeat_back_addr":"172.21.15.170:6815/47584024","heartbeat_front_addr":"172.21.15.170:6813/47584024","state":["exists","up"]},{"osd":6,"uuid":"46e00786-a5d2-48fe-b750-89beb0d63194","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":63,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6816","nonce":814825309},{"type":"v1","addr":"172.21.15.170:6817","nonce":814825309}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6818","nonce":814825309},{"type":"v1","addr":"172.21.15.170:6819","nonce":814825309}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6822","nonce":814825309},{"type":"v1","addr":"172.21.15.170:6823","nonce":814825309}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6820","nonce":814825309},{"type":"v1","addr":"172.21.15.170:6821","nonce":814825309}]},"public_addr":"172.21.15.170:6817/814825309","cluster_addr":"172.21.15.170:6819/814825309","heartbeat_back_addr":"172.21.15.170:6823/814825309","heartbeat_front_addr":"172.21.15.170:6821/814825309","state":["exists","up"]},{"osd":7,"uuid":"56567866-297b-457f-922a-7a53a1f3ef21","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":54,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6824","nonce":472398409},{"type":"v1","addr":"172.21.15.170:6825","nonce":472398409}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6826","nonce":472398409},{"type":"v1","addr":"172.21.15.170:6827","nonce":472398409}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6830","nonce":472398409},{"type":"v1","addr":"172.21.15.170:6831","nonce":472398409}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6828","nonce":472398409},{"type":"v1","addr":"172.21.15.170:6829","nonce":472398409}]},"public_addr":"172.21.15.170:6825/472398409","cluster_addr":"172.21.15.170:6827/472398409","heartbeat_back_addr":"172.21.15.170:6831/472398409","heartbeat_front_addr":"172.21.15.170:6829/472398409","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-10T00:48:30.232907+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-10T00:48:56.221202+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-10T00:49:22.529778+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-10T00:49:52.090754+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-10T00:50:16.772217+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-10T00:50:42.035375+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-10T00:51:09.412164+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-10T00:51:35.931598+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.106:0/3982476751":"2024-03-11T00:47:00.732937+0000","172.21.15.106:6801/2449347734":"2024-03-11T00:47:00.732937+0000","172.21.15.106:6800/2449347734":"2024-03-11T00:47:00.732937+0000","172.21.15.106:0/3555678354":"2024-03-11T00:47:00.732937+0000","172.21.15.106:6801/1073099015":"2024-03-11T00:46:33.968987+0000","172.21.15.106:0/4150429375":"2024-03-11T00:46:33.968987+0000","172.21.15.106:6800/1073099015":"2024-03-11T00:46:33.968987+0000","172.21.15.106:0/232887326":"2024-03-11T00:46:33.968987+0000","172.21.15.106:0/4228752537":"2024-03-11T00:46:33.968987+0000","172.21.15.106:0/3721367154":"2024-03-11T00:47:00.732937+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-10T00:52:33.524 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:52:33 smithi170 ceph-mon[27620]: from='client.? 172.21.15.106:0/1675011020' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-10T00:52:33.588 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:52:33 smithi106 ceph-mon[25742]: from='client.? 172.21.15.106:0/1675011020' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-10T00:52:33.588 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:52:33 smithi106 ceph-mon[33460]: from='client.? 172.21.15.106:0/1675011020' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-10T00:52:33.864 INFO:journalctl@ceph.mgr.y.smithi106.stdout:Mar 10 00:52:33 smithi106 ceph-75c37498-de77-11ee-95c7-87774f69a715-mgr-y[25997]: 2024-03-10T00:52:33.731+0000 7fd00781c200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-10T00:52:33.865 INFO:journalctl@ceph.mgr.y.smithi106.stdout:Mar 10 00:52:33 smithi106 ceph-75c37498-de77-11ee-95c7-87774f69a715-mgr-y[25997]: 2024-03-10T00:52:33.865+0000 7fd00781c200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-10T00:52:34.024 INFO:journalctl@ceph.mgr.x.smithi170.stdout:Mar 10 00:52:33 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-mgr-x[30157]: 2024-03-10T00:52:33.758+0000 7f1a88cc8200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-10T00:52:34.024 INFO:journalctl@ceph.mgr.x.smithi170.stdout:Mar 10 00:52:33 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-mgr-x[30157]: 2024-03-10T00:52:33.890+0000 7f1a88cc8200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-10T00:52:34.061 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:831bc188fa69f143bbba69ac76119e784500e8e3 shell --fsid 75c37498-de77-11ee-95c7-87774f69a715 -- ceph tell osd.0 flush_pg_stats 2024-03-10T00:52:34.062 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:831bc188fa69f143bbba69ac76119e784500e8e3 shell --fsid 75c37498-de77-11ee-95c7-87774f69a715 -- ceph tell osd.1 flush_pg_stats 2024-03-10T00:52:34.062 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:831bc188fa69f143bbba69ac76119e784500e8e3 shell --fsid 75c37498-de77-11ee-95c7-87774f69a715 -- ceph tell osd.2 flush_pg_stats 2024-03-10T00:52:34.063 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:831bc188fa69f143bbba69ac76119e784500e8e3 shell --fsid 75c37498-de77-11ee-95c7-87774f69a715 -- ceph tell osd.3 flush_pg_stats 2024-03-10T00:52:34.063 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:831bc188fa69f143bbba69ac76119e784500e8e3 shell --fsid 75c37498-de77-11ee-95c7-87774f69a715 -- ceph tell osd.4 flush_pg_stats 2024-03-10T00:52:34.064 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:831bc188fa69f143bbba69ac76119e784500e8e3 shell --fsid 75c37498-de77-11ee-95c7-87774f69a715 -- ceph tell osd.5 flush_pg_stats 2024-03-10T00:52:34.064 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:831bc188fa69f143bbba69ac76119e784500e8e3 shell --fsid 75c37498-de77-11ee-95c7-87774f69a715 -- ceph tell osd.6 flush_pg_stats 2024-03-10T00:52:34.064 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:831bc188fa69f143bbba69ac76119e784500e8e3 shell --fsid 75c37498-de77-11ee-95c7-87774f69a715 -- ceph tell osd.7 flush_pg_stats 2024-03-10T00:52:34.442 INFO:journalctl@ceph.mgr.y.smithi106.stdout:Mar 10 00:52:34 smithi106 ceph-75c37498-de77-11ee-95c7-87774f69a715-mgr-y[25997]: 2024-03-10T00:52:34.288+0000 7fd00781c200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-10T00:52:34.509 INFO:journalctl@ceph.mgr.x.smithi170.stdout:Mar 10 00:52:34 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-mgr-x[30157]: 2024-03-10T00:52:34.226+0000 7f1a88cc8200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-10T00:52:34.509 INFO:journalctl@ceph.mgr.x.smithi170.stdout:Mar 10 00:52:34 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-mgr-x[30157]: 2024-03-10T00:52:34.315+0000 7f1a88cc8200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-10T00:52:34.699 INFO:journalctl@ceph.mgr.y.smithi106.stdout:Mar 10 00:52:34 smithi106 ceph-75c37498-de77-11ee-95c7-87774f69a715-mgr-y[25997]: 2024-03-10T00:52:34.443+0000 7fd00781c200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-10T00:52:34.700 INFO:journalctl@ceph.mgr.y.smithi106.stdout:Mar 10 00:52:34 smithi106 ceph-75c37498-de77-11ee-95c7-87774f69a715-mgr-y[25997]: 2024-03-10T00:52:34.700+0000 7fd00781c200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-10T00:52:34.774 INFO:journalctl@ceph.mgr.x.smithi170.stdout:Mar 10 00:52:34 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-mgr-x[30157]: 2024-03-10T00:52:34.511+0000 7f1a88cc8200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-10T00:52:34.774 INFO:journalctl@ceph.mgr.x.smithi170.stdout:Mar 10 00:52:34 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-mgr-x[30157]: 2024-03-10T00:52:34.697+0000 7f1a88cc8200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-10T00:52:35.088 INFO:journalctl@ceph.mgr.y.smithi106.stdout:Mar 10 00:52:34 smithi106 ceph-75c37498-de77-11ee-95c7-87774f69a715-mgr-y[25997]: 2024-03-10T00:52:34.904+0000 7fd00781c200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-10T00:52:35.131 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/75c37498-de77-11ee-95c7-87774f69a715/mon.a/config 2024-03-10T00:52:35.132 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/75c37498-de77-11ee-95c7-87774f69a715/mon.a/config 2024-03-10T00:52:35.134 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/75c37498-de77-11ee-95c7-87774f69a715/mon.a/config 2024-03-10T00:52:35.134 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/75c37498-de77-11ee-95c7-87774f69a715/mon.a/config 2024-03-10T00:52:35.134 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/75c37498-de77-11ee-95c7-87774f69a715/mon.a/config 2024-03-10T00:52:35.137 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/75c37498-de77-11ee-95c7-87774f69a715/mon.a/config 2024-03-10T00:52:35.146 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/75c37498-de77-11ee-95c7-87774f69a715/mon.a/config 2024-03-10T00:52:35.204 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/75c37498-de77-11ee-95c7-87774f69a715/mon.a/config 2024-03-10T00:52:35.242 INFO:journalctl@ceph.mgr.x.smithi170.stdout:Mar 10 00:52:34 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-mgr-x[30157]: 2024-03-10T00:52:34.982+0000 7f1a88cc8200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-10T00:52:35.445 INFO:journalctl@ceph.mgr.y.smithi106.stdout:Mar 10 00:52:35 smithi106 ceph-75c37498-de77-11ee-95c7-87774f69a715-mgr-y[25997]: 2024-03-10T00:52:35.247+0000 7fd00781c200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-10T00:52:35.524 INFO:journalctl@ceph.mgr.x.smithi170.stdout:Mar 10 00:52:35 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-mgr-x[30157]: 2024-03-10T00:52:35.244+0000 7f1a88cc8200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-10T00:52:35.524 INFO:journalctl@ceph.mgr.x.smithi170.stdout:Mar 10 00:52:35 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-mgr-x[30157]: 2024-03-10T00:52:35.480+0000 7f1a88cc8200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-10T00:52:35.695 INFO:journalctl@ceph.mgr.y.smithi106.stdout:Mar 10 00:52:35 smithi106 ceph-75c37498-de77-11ee-95c7-87774f69a715-mgr-y[25997]: 2024-03-10T00:52:35.519+0000 7fd00781c200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-10T00:52:36.088 INFO:journalctl@ceph.mgr.y.smithi106.stdout:Mar 10 00:52:35 smithi106 ceph-75c37498-de77-11ee-95c7-87774f69a715-mgr-y[25997]: 2024-03-10T00:52:35.766+0000 7fd00781c200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-10T00:52:36.774 INFO:journalctl@ceph.mgr.x.smithi170.stdout:Mar 10 00:52:36 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-mgr-x[30157]: 2024-03-10T00:52:36.281+0000 7f1a88cc8200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-10T00:52:36.838 INFO:journalctl@ceph.mgr.y.smithi106.stdout:Mar 10 00:52:36 smithi106 ceph-75c37498-de77-11ee-95c7-87774f69a715-mgr-y[25997]: 2024-03-10T00:52:36.615+0000 7fd00781c200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-10T00:52:37.171 INFO:journalctl@ceph.mgr.x.smithi170.stdout:Mar 10 00:52:36 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-mgr-x[30157]: 2024-03-10T00:52:36.855+0000 7f1a88cc8200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-10T00:52:37.171 INFO:journalctl@ceph.mgr.x.smithi170.stdout:Mar 10 00:52:36 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-mgr-x[30157]: 2024-03-10T00:52:36.954+0000 7f1a88cc8200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-10T00:52:37.172 INFO:journalctl@ceph.mgr.x.smithi170.stdout:Mar 10 00:52:37 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-mgr-x[30157]: 2024-03-10T00:52:37.173+0000 7f1a88cc8200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-10T00:52:37.244 INFO:journalctl@ceph.mgr.y.smithi106.stdout:Mar 10 00:52:37 smithi106 ceph-75c37498-de77-11ee-95c7-87774f69a715-mgr-y[25997]: 2024-03-10T00:52:37.206+0000 7fd00781c200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-10T00:52:37.524 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:52:37 smithi170 ceph-mon[27620]: Standby manager daemon x restarted 2024-03-10T00:52:37.524 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:52:37 smithi170 ceph-mon[27620]: Standby manager daemon x started 2024-03-10T00:52:37.524 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:52:37 smithi170 ceph-mon[27620]: from='mgr.? 172.21.15.170:0/2213659458' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-03-10T00:52:37.525 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:52:37 smithi170 ceph-mon[27620]: from='mgr.? 172.21.15.170:0/2213659458' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-10T00:52:37.525 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:52:37 smithi170 ceph-mon[27620]: from='mgr.? 172.21.15.170:0/2213659458' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-03-10T00:52:37.525 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:52:37 smithi170 ceph-mon[27620]: from='mgr.? 172.21.15.170:0/2213659458' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-10T00:52:37.525 INFO:journalctl@ceph.mgr.x.smithi170.stdout:Mar 10 00:52:37 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-mgr-x[30157]: [10/Mar/2024:00:52:37] ENGINE Bus STARTING 2024-03-10T00:52:37.525 INFO:journalctl@ceph.mgr.x.smithi170.stdout:Mar 10 00:52:37 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-mgr-x[30157]: CherryPy Checker: 2024-03-10T00:52:37.525 INFO:journalctl@ceph.mgr.x.smithi170.stdout:Mar 10 00:52:37 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-mgr-x[30157]: The Application mounted at '' has an empty config. 2024-03-10T00:52:37.525 INFO:journalctl@ceph.mgr.x.smithi170.stdout:Mar 10 00:52:37 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-mgr-x[30157]: 2024-03-10T00:52:37.525 INFO:journalctl@ceph.mgr.x.smithi170.stdout:Mar 10 00:52:37 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-mgr-x[30157]: [10/Mar/2024:00:52:37] ENGINE Serving on http://:::9283 2024-03-10T00:52:37.526 INFO:journalctl@ceph.mgr.x.smithi170.stdout:Mar 10 00:52:37 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-mgr-x[30157]: [10/Mar/2024:00:52:37] ENGINE Bus STARTED 2024-03-10T00:52:37.531 INFO:journalctl@ceph.mgr.y.smithi106.stdout:Mar 10 00:52:37 smithi106 ceph-75c37498-de77-11ee-95c7-87774f69a715-mgr-y[25997]: 2024-03-10T00:52:37.312+0000 7fd00781c200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-10T00:52:37.531 INFO:journalctl@ceph.mgr.y.smithi106.stdout:Mar 10 00:52:37 smithi106 ceph-75c37498-de77-11ee-95c7-87774f69a715-mgr-y[25997]: 2024-03-10T00:52:37.532+0000 7fd00781c200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-10T00:52:37.531 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:52:37 smithi106 ceph-mon[25742]: Standby manager daemon x restarted 2024-03-10T00:52:37.531 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:52:37 smithi106 ceph-mon[25742]: Standby manager daemon x started 2024-03-10T00:52:37.532 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:52:37 smithi106 ceph-mon[25742]: from='mgr.? 172.21.15.170:0/2213659458' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-03-10T00:52:37.532 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:52:37 smithi106 ceph-mon[25742]: from='mgr.? 172.21.15.170:0/2213659458' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-10T00:52:37.532 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:52:37 smithi106 ceph-mon[25742]: from='mgr.? 172.21.15.170:0/2213659458' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-03-10T00:52:37.532 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:52:37 smithi106 ceph-mon[25742]: from='mgr.? 172.21.15.170:0/2213659458' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-10T00:52:37.532 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:52:37 smithi106 ceph-mon[33460]: Standby manager daemon x restarted 2024-03-10T00:52:37.532 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:52:37 smithi106 ceph-mon[33460]: Standby manager daemon x started 2024-03-10T00:52:37.532 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:52:37 smithi106 ceph-mon[33460]: from='mgr.? 172.21.15.170:0/2213659458' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-03-10T00:52:37.532 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:52:37 smithi106 ceph-mon[33460]: from='mgr.? 172.21.15.170:0/2213659458' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-10T00:52:37.532 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:52:37 smithi106 ceph-mon[33460]: from='mgr.? 172.21.15.170:0/2213659458' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-03-10T00:52:37.532 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:52:37 smithi106 ceph-mon[33460]: from='mgr.? 172.21.15.170:0/2213659458' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-10T00:52:37.838 INFO:journalctl@ceph.mgr.y.smithi106.stdout:Mar 10 00:52:37 smithi106 ceph-75c37498-de77-11ee-95c7-87774f69a715-mgr-y[25997]: [10/Mar/2024:00:52:37] ENGINE Bus STARTING 2024-03-10T00:52:37.838 INFO:journalctl@ceph.mgr.y.smithi106.stdout:Mar 10 00:52:37 smithi106 ceph-75c37498-de77-11ee-95c7-87774f69a715-mgr-y[25997]: CherryPy Checker: 2024-03-10T00:52:37.839 INFO:journalctl@ceph.mgr.y.smithi106.stdout:Mar 10 00:52:37 smithi106 ceph-75c37498-de77-11ee-95c7-87774f69a715-mgr-y[25997]: The Application mounted at '' has an empty config. 2024-03-10T00:52:37.839 INFO:journalctl@ceph.mgr.y.smithi106.stdout:Mar 10 00:52:37 smithi106 ceph-75c37498-de77-11ee-95c7-87774f69a715-mgr-y[25997]: 2024-03-10T00:52:37.839 INFO:journalctl@ceph.mgr.y.smithi106.stdout:Mar 10 00:52:37 smithi106 ceph-75c37498-de77-11ee-95c7-87774f69a715-mgr-y[25997]: [10/Mar/2024:00:52:37] ENGINE Serving on http://:::9283 2024-03-10T00:52:37.839 INFO:journalctl@ceph.mgr.y.smithi106.stdout:Mar 10 00:52:37 smithi106 ceph-75c37498-de77-11ee-95c7-87774f69a715-mgr-y[25997]: [10/Mar/2024:00:52:37] ENGINE Bus STARTED 2024-03-10T00:52:38.524 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:52:38 smithi170 ceph-mon[27620]: mgrmap e18: y(active, since 5m), standbys: x 2024-03-10T00:52:38.525 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:52:38 smithi170 ceph-mon[27620]: Active manager daemon y restarted 2024-03-10T00:52:38.525 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:52:38 smithi170 ceph-mon[27620]: Activating manager daemon y 2024-03-10T00:52:38.525 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:52:38 smithi170 ceph-mon[27620]: osdmap e69: 8 total, 8 up, 8 in 2024-03-10T00:52:38.525 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:52:38 smithi170 ceph-mon[27620]: mgrmap e19: y(active, starting, since 0.00902359s), standbys: x 2024-03-10T00:52:38.525 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:52:38 smithi170 ceph-mon[27620]: from='mgr.14634 172.21.15.106:0/2720500564' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-10T00:52:38.525 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:52:38 smithi170 ceph-mon[27620]: from='mgr.14634 172.21.15.106:0/2720500564' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-10T00:52:38.525 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:52:38 smithi170 ceph-mon[27620]: from='mgr.14634 172.21.15.106:0/2720500564' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-10T00:52:38.525 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:52:38 smithi170 ceph-mon[27620]: from='mgr.14634 172.21.15.106:0/2720500564' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-10T00:52:38.525 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:52:38 smithi170 ceph-mon[27620]: from='mgr.14634 172.21.15.106:0/2720500564' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-10T00:52:38.525 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:52:38 smithi170 ceph-mon[27620]: from='mgr.14634 172.21.15.106:0/2720500564' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-10T00:52:38.525 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:52:38 smithi170 ceph-mon[27620]: from='mgr.14634 172.21.15.106:0/2720500564' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-10T00:52:38.525 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:52:38 smithi170 ceph-mon[27620]: from='mgr.14634 172.21.15.106:0/2720500564' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-10T00:52:38.525 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:52:38 smithi170 ceph-mon[27620]: from='mgr.14634 172.21.15.106:0/2720500564' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-10T00:52:38.526 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:52:38 smithi170 ceph-mon[27620]: from='mgr.14634 172.21.15.106:0/2720500564' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-10T00:52:38.526 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:52:38 smithi170 ceph-mon[27620]: from='mgr.14634 172.21.15.106:0/2720500564' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-10T00:52:38.526 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:52:38 smithi170 ceph-mon[27620]: from='mgr.14634 172.21.15.106:0/2720500564' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-10T00:52:38.526 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:52:38 smithi170 ceph-mon[27620]: from='mgr.14634 172.21.15.106:0/2720500564' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-10T00:52:38.526 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:52:38 smithi170 ceph-mon[27620]: from='mgr.14634 172.21.15.106:0/2720500564' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-10T00:52:38.526 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:52:38 smithi170 ceph-mon[27620]: from='mgr.14634 172.21.15.106:0/2720500564' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-10T00:52:38.526 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:52:38 smithi170 ceph-mon[27620]: from='mgr.14634 172.21.15.106:0/2720500564' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-10T00:52:38.526 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:52:38 smithi170 ceph-mon[27620]: Manager daemon y is now available 2024-03-10T00:52:38.526 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:52:38 smithi170 ceph-mon[27620]: from='mgr.14634 172.21.15.106:0/2720500564' entity='mgr.y' 2024-03-10T00:52:38.526 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:52:38 smithi170 ceph-mon[27620]: from='mgr.14634 172.21.15.106:0/2720500564' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-10T00:52:38.526 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:52:38 smithi170 ceph-mon[27620]: from='mgr.14634 172.21.15.106:0/2720500564' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-10T00:52:38.526 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:52:38 smithi170 ceph-mon[27620]: from='mgr.14634 172.21.15.106:0/2720500564' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi170", "name": "osd_memory_target"}]: dispatch 2024-03-10T00:52:38.527 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:52:38 smithi170 ceph-mon[27620]: from='mgr.14634 172.21.15.106:0/2720500564' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi106", "name": "osd_memory_target"}]: dispatch 2024-03-10T00:52:38.527 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:52:38 smithi170 ceph-mon[27620]: from='mgr.14634 172.21.15.106:0/2720500564' entity='mgr.y' cmd='[{"prefix": "config rm", "who": "osd/host:smithi170", "name": "osd_memory_target"}]': finished 2024-03-10T00:52:38.527 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:52:38 smithi170 ceph-mon[27620]: from='mgr.14634 172.21.15.106:0/2720500564' entity='mgr.y' cmd='[{"prefix": "config rm", "who": "osd/host:smithi106", "name": "osd_memory_target"}]': finished 2024-03-10T00:52:38.527 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:52:38 smithi170 ceph-mon[27620]: from='mgr.14634 172.21.15.106:0/2720500564' entity='mgr.y' 2024-03-10T00:52:38.527 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:52:38 smithi170 ceph-mon[27620]: from='mgr.14634 172.21.15.106:0/2720500564' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-10T00:52:38.527 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:52:38 smithi170 ceph-mon[27620]: from='mgr.14634 172.21.15.106:0/2720500564' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-10T00:52:38.527 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:52:38 smithi170 ceph-mon[27620]: from='mgr.14634 172.21.15.106:0/2720500564' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-10T00:52:38.527 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:52:38 smithi170 ceph-mon[27620]: from='mgr.14634 172.21.15.106:0/2720500564' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-10T00:52:38.527 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:52:38 smithi170 ceph-mon[27620]: from='mgr.14634 172.21.15.106:0/2720500564' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-10T00:52:38.589 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:52:38 smithi106 ceph-mon[33460]: mgrmap e18: y(active, since 5m), standbys: x 2024-03-10T00:52:38.589 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:52:38 smithi106 ceph-mon[33460]: Active manager daemon y restarted 2024-03-10T00:52:38.589 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:52:38 smithi106 ceph-mon[33460]: Activating manager daemon y 2024-03-10T00:52:38.589 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:52:38 smithi106 ceph-mon[33460]: osdmap e69: 8 total, 8 up, 8 in 2024-03-10T00:52:38.589 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:52:38 smithi106 ceph-mon[33460]: mgrmap e19: y(active, starting, since 0.00902359s), standbys: x 2024-03-10T00:52:38.590 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:52:38 smithi106 ceph-mon[33460]: from='mgr.14634 172.21.15.106:0/2720500564' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-10T00:52:38.590 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:52:38 smithi106 ceph-mon[33460]: from='mgr.14634 172.21.15.106:0/2720500564' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-10T00:52:38.590 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:52:38 smithi106 ceph-mon[33460]: from='mgr.14634 172.21.15.106:0/2720500564' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-10T00:52:38.590 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:52:38 smithi106 ceph-mon[33460]: from='mgr.14634 172.21.15.106:0/2720500564' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-10T00:52:38.590 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:52:38 smithi106 ceph-mon[33460]: from='mgr.14634 172.21.15.106:0/2720500564' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-10T00:52:38.590 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:52:38 smithi106 ceph-mon[33460]: from='mgr.14634 172.21.15.106:0/2720500564' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-10T00:52:38.590 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:52:38 smithi106 ceph-mon[33460]: from='mgr.14634 172.21.15.106:0/2720500564' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-10T00:52:38.590 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:52:38 smithi106 ceph-mon[33460]: from='mgr.14634 172.21.15.106:0/2720500564' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-10T00:52:38.591 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:52:38 smithi106 ceph-mon[33460]: from='mgr.14634 172.21.15.106:0/2720500564' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-10T00:52:38.591 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:52:38 smithi106 ceph-mon[33460]: from='mgr.14634 172.21.15.106:0/2720500564' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-10T00:52:38.591 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:52:38 smithi106 ceph-mon[33460]: from='mgr.14634 172.21.15.106:0/2720500564' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-10T00:52:38.591 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:52:38 smithi106 ceph-mon[33460]: from='mgr.14634 172.21.15.106:0/2720500564' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-10T00:52:38.591 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:52:38 smithi106 ceph-mon[33460]: from='mgr.14634 172.21.15.106:0/2720500564' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-10T00:52:38.591 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:52:38 smithi106 ceph-mon[33460]: from='mgr.14634 172.21.15.106:0/2720500564' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-10T00:52:38.591 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:52:38 smithi106 ceph-mon[33460]: from='mgr.14634 172.21.15.106:0/2720500564' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-10T00:52:38.591 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:52:38 smithi106 ceph-mon[33460]: from='mgr.14634 172.21.15.106:0/2720500564' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-10T00:52:38.592 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:52:38 smithi106 ceph-mon[33460]: Manager daemon y is now available 2024-03-10T00:52:38.592 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:52:38 smithi106 ceph-mon[33460]: from='mgr.14634 172.21.15.106:0/2720500564' entity='mgr.y' 2024-03-10T00:52:38.592 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:52:38 smithi106 ceph-mon[33460]: from='mgr.14634 172.21.15.106:0/2720500564' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-10T00:52:38.592 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:52:38 smithi106 ceph-mon[33460]: from='mgr.14634 172.21.15.106:0/2720500564' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-10T00:52:38.592 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:52:38 smithi106 ceph-mon[33460]: from='mgr.14634 172.21.15.106:0/2720500564' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi170", "name": "osd_memory_target"}]: dispatch 2024-03-10T00:52:38.592 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:52:38 smithi106 ceph-mon[33460]: from='mgr.14634 172.21.15.106:0/2720500564' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi106", "name": "osd_memory_target"}]: dispatch 2024-03-10T00:52:38.592 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:52:38 smithi106 ceph-mon[33460]: from='mgr.14634 172.21.15.106:0/2720500564' entity='mgr.y' cmd='[{"prefix": "config rm", "who": "osd/host:smithi170", "name": "osd_memory_target"}]': finished 2024-03-10T00:52:38.593 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:52:38 smithi106 ceph-mon[25742]: mgrmap e18: y(active, since 5m), standbys: x 2024-03-10T00:52:38.593 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:52:38 smithi106 ceph-mon[25742]: Active manager daemon y restarted 2024-03-10T00:52:38.593 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:52:38 smithi106 ceph-mon[25742]: Activating manager daemon y 2024-03-10T00:52:38.593 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:52:38 smithi106 ceph-mon[25742]: osdmap e69: 8 total, 8 up, 8 in 2024-03-10T00:52:38.593 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:52:38 smithi106 ceph-mon[25742]: mgrmap e19: y(active, starting, since 0.00902359s), standbys: x 2024-03-10T00:52:38.594 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:52:38 smithi106 ceph-mon[25742]: from='mgr.14634 172.21.15.106:0/2720500564' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-10T00:52:38.594 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:52:38 smithi106 ceph-mon[25742]: from='mgr.14634 172.21.15.106:0/2720500564' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-10T00:52:38.594 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:52:38 smithi106 ceph-mon[25742]: from='mgr.14634 172.21.15.106:0/2720500564' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-10T00:52:38.594 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:52:38 smithi106 ceph-mon[25742]: from='mgr.14634 172.21.15.106:0/2720500564' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-10T00:52:38.594 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:52:38 smithi106 ceph-mon[25742]: from='mgr.14634 172.21.15.106:0/2720500564' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-10T00:52:38.594 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:52:38 smithi106 ceph-mon[25742]: from='mgr.14634 172.21.15.106:0/2720500564' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-10T00:52:38.594 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:52:38 smithi106 ceph-mon[25742]: from='mgr.14634 172.21.15.106:0/2720500564' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-10T00:52:38.595 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:52:38 smithi106 ceph-mon[25742]: from='mgr.14634 172.21.15.106:0/2720500564' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-10T00:52:38.595 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:52:38 smithi106 ceph-mon[25742]: from='mgr.14634 172.21.15.106:0/2720500564' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-10T00:52:38.595 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:52:38 smithi106 ceph-mon[25742]: from='mgr.14634 172.21.15.106:0/2720500564' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-10T00:52:38.595 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:52:38 smithi106 ceph-mon[25742]: from='mgr.14634 172.21.15.106:0/2720500564' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-10T00:52:38.595 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:52:38 smithi106 ceph-mon[25742]: from='mgr.14634 172.21.15.106:0/2720500564' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-10T00:52:38.595 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:52:38 smithi106 ceph-mon[25742]: from='mgr.14634 172.21.15.106:0/2720500564' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-10T00:52:38.595 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:52:38 smithi106 ceph-mon[25742]: from='mgr.14634 172.21.15.106:0/2720500564' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-10T00:52:38.595 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:52:38 smithi106 ceph-mon[25742]: from='mgr.14634 172.21.15.106:0/2720500564' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-10T00:52:38.595 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:52:38 smithi106 ceph-mon[25742]: from='mgr.14634 172.21.15.106:0/2720500564' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-10T00:52:38.596 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:52:38 smithi106 ceph-mon[25742]: Manager daemon y is now available 2024-03-10T00:52:38.596 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:52:38 smithi106 ceph-mon[25742]: from='mgr.14634 172.21.15.106:0/2720500564' entity='mgr.y' 2024-03-10T00:52:38.596 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:52:38 smithi106 ceph-mon[25742]: from='mgr.14634 172.21.15.106:0/2720500564' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-10T00:52:38.596 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:52:38 smithi106 ceph-mon[25742]: from='mgr.14634 172.21.15.106:0/2720500564' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-10T00:52:38.596 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:52:38 smithi106 ceph-mon[25742]: from='mgr.14634 172.21.15.106:0/2720500564' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi170", "name": "osd_memory_target"}]: dispatch 2024-03-10T00:52:38.596 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:52:38 smithi106 ceph-mon[25742]: from='mgr.14634 172.21.15.106:0/2720500564' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi106", "name": "osd_memory_target"}]: dispatch 2024-03-10T00:52:38.596 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:52:38 smithi106 ceph-mon[25742]: from='mgr.14634 172.21.15.106:0/2720500564' entity='mgr.y' cmd='[{"prefix": "config rm", "who": "osd/host:smithi170", "name": "osd_memory_target"}]': finished 2024-03-10T00:52:38.597 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:52:38 smithi106 ceph-mon[25742]: from='mgr.14634 172.21.15.106:0/2720500564' entity='mgr.y' cmd='[{"prefix": "config rm", "who": "osd/host:smithi106", "name": "osd_memory_target"}]': finished 2024-03-10T00:52:38.597 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:52:38 smithi106 ceph-mon[25742]: from='mgr.14634 172.21.15.106:0/2720500564' entity='mgr.y' 2024-03-10T00:52:38.597 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:52:38 smithi106 ceph-mon[25742]: from='mgr.14634 172.21.15.106:0/2720500564' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-10T00:52:38.597 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:52:38 smithi106 ceph-mon[25742]: from='mgr.14634 172.21.15.106:0/2720500564' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-10T00:52:38.597 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:52:38 smithi106 ceph-mon[25742]: from='mgr.14634 172.21.15.106:0/2720500564' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-10T00:52:38.597 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:52:38 smithi106 ceph-mon[25742]: from='mgr.14634 172.21.15.106:0/2720500564' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-10T00:52:38.597 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:52:38 smithi106 ceph-mon[25742]: from='mgr.14634 172.21.15.106:0/2720500564' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-10T00:52:38.598 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:52:38 smithi106 ceph-mon[33460]: from='mgr.14634 172.21.15.106:0/2720500564' entity='mgr.y' cmd='[{"prefix": "config rm", "who": "osd/host:smithi106", "name": "osd_memory_target"}]': finished 2024-03-10T00:52:38.598 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:52:38 smithi106 ceph-mon[33460]: from='mgr.14634 172.21.15.106:0/2720500564' entity='mgr.y' 2024-03-10T00:52:38.598 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:52:38 smithi106 ceph-mon[33460]: from='mgr.14634 172.21.15.106:0/2720500564' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-10T00:52:38.598 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:52:38 smithi106 ceph-mon[33460]: from='mgr.14634 172.21.15.106:0/2720500564' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-10T00:52:38.599 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:52:38 smithi106 ceph-mon[33460]: from='mgr.14634 172.21.15.106:0/2720500564' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-10T00:52:38.599 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:52:38 smithi106 ceph-mon[33460]: from='mgr.14634 172.21.15.106:0/2720500564' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-10T00:52:38.599 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:52:38 smithi106 ceph-mon[33460]: from='mgr.14634 172.21.15.106:0/2720500564' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-10T00:52:39.024 INFO:journalctl@ceph.iscsi.iscsi.a.smithi170.stdout:Mar 10 00:52:38 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-iscsi-iscsi-a[50134]: debug there is no tcmu-runner data available 2024-03-10T00:52:39.524 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:52:39 smithi170 ceph-mon[27620]: Updating smithi106:/etc/ceph/ceph.conf 2024-03-10T00:52:39.524 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:52:39 smithi170 ceph-mon[27620]: Updating smithi170:/etc/ceph/ceph.conf 2024-03-10T00:52:39.524 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:52:39 smithi170 ceph-mon[27620]: Updating smithi106:/var/lib/ceph/75c37498-de77-11ee-95c7-87774f69a715/config/ceph.conf 2024-03-10T00:52:39.524 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:52:39 smithi170 ceph-mon[27620]: Updating smithi170:/var/lib/ceph/75c37498-de77-11ee-95c7-87774f69a715/config/ceph.conf 2024-03-10T00:52:39.524 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:52:39 smithi170 ceph-mon[27620]: mgrmap e20: y(active, since 1.0129s), standbys: x 2024-03-10T00:52:39.524 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:52:39 smithi170 ceph-mon[27620]: from='mgr.14634 172.21.15.106:0/2720500564' entity='mgr.y' 2024-03-10T00:52:39.525 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:52:39 smithi170 ceph-mon[27620]: from='mgr.14634 172.21.15.106:0/2720500564' entity='mgr.y' 2024-03-10T00:52:39.525 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:52:39 smithi170 ceph-mon[27620]: from='mgr.14634 172.21.15.106:0/2720500564' entity='mgr.y' 2024-03-10T00:52:39.525 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:52:39 smithi170 ceph-mon[27620]: from='mgr.14634 172.21.15.106:0/2720500564' entity='mgr.y' 2024-03-10T00:52:39.525 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:52:39 smithi170 ceph-mon[27620]: from='mgr.14634 172.21.15.106:0/2720500564' entity='mgr.y' 2024-03-10T00:52:39.525 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:52:39 smithi170 ceph-mon[27620]: from='mgr.14634 172.21.15.106:0/2720500564' entity='mgr.y' 2024-03-10T00:52:39.588 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:52:39 smithi106 ceph-mon[25742]: Updating smithi106:/etc/ceph/ceph.conf 2024-03-10T00:52:39.588 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:52:39 smithi106 ceph-mon[25742]: Updating smithi170:/etc/ceph/ceph.conf 2024-03-10T00:52:39.589 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:52:39 smithi106 ceph-mon[25742]: Updating smithi106:/var/lib/ceph/75c37498-de77-11ee-95c7-87774f69a715/config/ceph.conf 2024-03-10T00:52:39.589 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:52:39 smithi106 ceph-mon[25742]: Updating smithi170:/var/lib/ceph/75c37498-de77-11ee-95c7-87774f69a715/config/ceph.conf 2024-03-10T00:52:39.589 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:52:39 smithi106 ceph-mon[25742]: mgrmap e20: y(active, since 1.0129s), standbys: x 2024-03-10T00:52:39.589 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:52:39 smithi106 ceph-mon[25742]: from='mgr.14634 172.21.15.106:0/2720500564' entity='mgr.y' 2024-03-10T00:52:39.589 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:52:39 smithi106 ceph-mon[25742]: from='mgr.14634 172.21.15.106:0/2720500564' entity='mgr.y' 2024-03-10T00:52:39.589 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:52:39 smithi106 ceph-mon[25742]: from='mgr.14634 172.21.15.106:0/2720500564' entity='mgr.y' 2024-03-10T00:52:39.589 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:52:39 smithi106 ceph-mon[25742]: from='mgr.14634 172.21.15.106:0/2720500564' entity='mgr.y' 2024-03-10T00:52:39.589 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:52:39 smithi106 ceph-mon[25742]: from='mgr.14634 172.21.15.106:0/2720500564' entity='mgr.y' 2024-03-10T00:52:39.589 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:52:39 smithi106 ceph-mon[25742]: from='mgr.14634 172.21.15.106:0/2720500564' entity='mgr.y' 2024-03-10T00:52:39.589 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:52:39 smithi106 ceph-mon[33460]: Updating smithi106:/etc/ceph/ceph.conf 2024-03-10T00:52:39.589 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:52:39 smithi106 ceph-mon[33460]: Updating smithi170:/etc/ceph/ceph.conf 2024-03-10T00:52:39.590 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:52:39 smithi106 ceph-mon[33460]: Updating smithi106:/var/lib/ceph/75c37498-de77-11ee-95c7-87774f69a715/config/ceph.conf 2024-03-10T00:52:39.590 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:52:39 smithi106 ceph-mon[33460]: Updating smithi170:/var/lib/ceph/75c37498-de77-11ee-95c7-87774f69a715/config/ceph.conf 2024-03-10T00:52:39.590 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:52:39 smithi106 ceph-mon[33460]: mgrmap e20: y(active, since 1.0129s), standbys: x 2024-03-10T00:52:39.590 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:52:39 smithi106 ceph-mon[33460]: from='mgr.14634 172.21.15.106:0/2720500564' entity='mgr.y' 2024-03-10T00:52:39.590 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:52:39 smithi106 ceph-mon[33460]: from='mgr.14634 172.21.15.106:0/2720500564' entity='mgr.y' 2024-03-10T00:52:39.590 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:52:39 smithi106 ceph-mon[33460]: from='mgr.14634 172.21.15.106:0/2720500564' entity='mgr.y' 2024-03-10T00:52:39.590 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:52:39 smithi106 ceph-mon[33460]: from='mgr.14634 172.21.15.106:0/2720500564' entity='mgr.y' 2024-03-10T00:52:39.590 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:52:39 smithi106 ceph-mon[33460]: from='mgr.14634 172.21.15.106:0/2720500564' entity='mgr.y' 2024-03-10T00:52:39.590 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:52:39 smithi106 ceph-mon[33460]: from='mgr.14634 172.21.15.106:0/2720500564' entity='mgr.y' 2024-03-10T00:52:40.524 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:52:40 smithi170 ceph-mon[27620]: [10/Mar/2024:00:52:38] ENGINE Bus STARTING 2024-03-10T00:52:40.524 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:52:40 smithi170 ceph-mon[27620]: Updating smithi106:/etc/ceph/ceph.client.admin.keyring 2024-03-10T00:52:40.524 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:52:40 smithi170 ceph-mon[27620]: Updating smithi170:/etc/ceph/ceph.client.admin.keyring 2024-03-10T00:52:40.524 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:52:40 smithi170 ceph-mon[27620]: [10/Mar/2024:00:52:38] ENGINE Serving on https://172.21.15.106:7151 2024-03-10T00:52:40.524 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:52:40 smithi170 ceph-mon[27620]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-10T00:52:40.525 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:52:40 smithi170 ceph-mon[27620]: [10/Mar/2024:00:52:38] ENGINE Serving on http://172.21.15.106:8765 2024-03-10T00:52:40.525 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:52:40 smithi170 ceph-mon[27620]: [10/Mar/2024:00:52:38] ENGINE Bus STARTED 2024-03-10T00:52:40.525 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:52:40 smithi170 ceph-mon[27620]: pgmap v3: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail 2024-03-10T00:52:40.525 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:52:40 smithi170 ceph-mon[27620]: Updating smithi106:/var/lib/ceph/75c37498-de77-11ee-95c7-87774f69a715/config/ceph.client.admin.keyring 2024-03-10T00:52:40.525 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:52:40 smithi170 ceph-mon[27620]: Updating smithi170:/var/lib/ceph/75c37498-de77-11ee-95c7-87774f69a715/config/ceph.client.admin.keyring 2024-03-10T00:52:40.525 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:52:40 smithi170 ceph-mon[27620]: Deploying daemon alertmanager.a on smithi106 2024-03-10T00:52:40.588 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:52:40 smithi106 ceph-mon[25742]: [10/Mar/2024:00:52:38] ENGINE Bus STARTING 2024-03-10T00:52:40.588 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:52:40 smithi106 ceph-mon[25742]: Updating smithi106:/etc/ceph/ceph.client.admin.keyring 2024-03-10T00:52:40.588 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:52:40 smithi106 ceph-mon[25742]: Updating smithi170:/etc/ceph/ceph.client.admin.keyring 2024-03-10T00:52:40.588 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:52:40 smithi106 ceph-mon[25742]: [10/Mar/2024:00:52:38] ENGINE Serving on https://172.21.15.106:7151 2024-03-10T00:52:40.588 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:52:40 smithi106 ceph-mon[25742]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-10T00:52:40.589 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:52:40 smithi106 ceph-mon[25742]: [10/Mar/2024:00:52:38] ENGINE Serving on http://172.21.15.106:8765 2024-03-10T00:52:40.589 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:52:40 smithi106 ceph-mon[25742]: [10/Mar/2024:00:52:38] ENGINE Bus STARTED 2024-03-10T00:52:40.589 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:52:40 smithi106 ceph-mon[25742]: pgmap v3: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail 2024-03-10T00:52:40.589 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:52:40 smithi106 ceph-mon[25742]: Updating smithi106:/var/lib/ceph/75c37498-de77-11ee-95c7-87774f69a715/config/ceph.client.admin.keyring 2024-03-10T00:52:40.589 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:52:40 smithi106 ceph-mon[25742]: Updating smithi170:/var/lib/ceph/75c37498-de77-11ee-95c7-87774f69a715/config/ceph.client.admin.keyring 2024-03-10T00:52:40.589 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:52:40 smithi106 ceph-mon[25742]: Deploying daemon alertmanager.a on smithi106 2024-03-10T00:52:40.589 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:52:40 smithi106 ceph-mon[33460]: [10/Mar/2024:00:52:38] ENGINE Bus STARTING 2024-03-10T00:52:40.589 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:52:40 smithi106 ceph-mon[33460]: Updating smithi106:/etc/ceph/ceph.client.admin.keyring 2024-03-10T00:52:40.589 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:52:40 smithi106 ceph-mon[33460]: Updating smithi170:/etc/ceph/ceph.client.admin.keyring 2024-03-10T00:52:40.590 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:52:40 smithi106 ceph-mon[33460]: [10/Mar/2024:00:52:38] ENGINE Serving on https://172.21.15.106:7151 2024-03-10T00:52:40.590 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:52:40 smithi106 ceph-mon[33460]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-10T00:52:40.590 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:52:40 smithi106 ceph-mon[33460]: [10/Mar/2024:00:52:38] ENGINE Serving on http://172.21.15.106:8765 2024-03-10T00:52:40.590 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:52:40 smithi106 ceph-mon[33460]: [10/Mar/2024:00:52:38] ENGINE Bus STARTED 2024-03-10T00:52:40.590 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:52:40 smithi106 ceph-mon[33460]: pgmap v3: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail 2024-03-10T00:52:40.590 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:52:40 smithi106 ceph-mon[33460]: Updating smithi106:/var/lib/ceph/75c37498-de77-11ee-95c7-87774f69a715/config/ceph.client.admin.keyring 2024-03-10T00:52:40.590 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:52:40 smithi106 ceph-mon[33460]: Updating smithi170:/var/lib/ceph/75c37498-de77-11ee-95c7-87774f69a715/config/ceph.client.admin.keyring 2024-03-10T00:52:40.590 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:52:40 smithi106 ceph-mon[33460]: Deploying daemon alertmanager.a on smithi106 2024-03-10T00:52:41.493 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:52:41 smithi106 ceph-mon[25742]: pgmap v4: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail 2024-03-10T00:52:41.494 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:52:41 smithi106 ceph-mon[25742]: mgrmap e21: y(active, since 3s), standbys: x 2024-03-10T00:52:41.494 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:52:41 smithi106 ceph-mon[33460]: pgmap v4: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail 2024-03-10T00:52:41.494 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:52:41 smithi106 ceph-mon[33460]: mgrmap e21: y(active, since 3s), standbys: x 2024-03-10T00:52:41.524 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:52:41 smithi170 ceph-mon[27620]: pgmap v4: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail 2024-03-10T00:52:41.524 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:52:41 smithi170 ceph-mon[27620]: mgrmap e21: y(active, since 3s), standbys: x 2024-03-10T00:52:41.838 INFO:journalctl@ceph.mgr.y.smithi106.stdout:Mar 10 00:52:41 smithi106 ceph-75c37498-de77-11ee-95c7-87774f69a715-mgr-y[25997]: ::ffff:172.21.15.170 - - [10/Mar/2024:00:52:41] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-10T00:52:43.471 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:52:43 smithi106 ceph-mon[25742]: pgmap v5: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail 2024-03-10T00:52:43.472 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:52:43 smithi106 ceph-mon[25742]: mgrmap e22: y(active, since 4s), standbys: x 2024-03-10T00:52:43.472 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:52:43 smithi106 ceph-mon[33460]: pgmap v5: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail 2024-03-10T00:52:43.472 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:52:43 smithi106 ceph-mon[33460]: mgrmap e22: y(active, since 4s), standbys: x 2024-03-10T00:52:43.524 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:52:43 smithi170 ceph-mon[27620]: pgmap v5: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail 2024-03-10T00:52:43.524 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:52:43 smithi170 ceph-mon[27620]: mgrmap e22: y(active, since 4s), standbys: x 2024-03-10T00:52:45.524 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:52:45 smithi170 ceph-mon[27620]: pgmap v6: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail 2024-03-10T00:52:45.565 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:52:45 smithi106 ceph-mon[25742]: pgmap v6: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail 2024-03-10T00:52:45.566 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:52:45 smithi106 ceph-mon[33460]: pgmap v6: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail 2024-03-10T00:52:47.134 INFO:teuthology.orchestra.run.smithi106.stdout:197568495636 2024-03-10T00:52:47.134 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:831bc188fa69f143bbba69ac76119e784500e8e3 shell --fsid 75c37498-de77-11ee-95c7-87774f69a715 -- ceph osd last-stat-seq osd.6 2024-03-10T00:52:47.436 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:52:47 smithi106 ceph-mon[25742]: pgmap v7: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 30 KiB/s rd, 0 B/s wr, 17 op/s 2024-03-10T00:52:47.437 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:52:47 smithi106 ceph-mon[33460]: pgmap v7: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 30 KiB/s rd, 0 B/s wr, 17 op/s 2024-03-10T00:52:47.524 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:52:47 smithi170 ceph-mon[27620]: pgmap v7: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 30 KiB/s rd, 0 B/s wr, 17 op/s 2024-03-10T00:52:48.176 INFO:teuthology.orchestra.run.smithi106.stdout:128849018911 2024-03-10T00:52:48.176 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:831bc188fa69f143bbba69ac76119e784500e8e3 shell --fsid 75c37498-de77-11ee-95c7-87774f69a715 -- ceph osd last-stat-seq osd.4 2024-03-10T00:52:48.408 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:52:48 smithi106 ceph-mon[33460]: pgmap v8: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 23 KiB/s rd, 0 B/s wr, 13 op/s 2024-03-10T00:52:48.409 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:52:48 smithi106 ceph-mon[25742]: pgmap v8: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 23 KiB/s rd, 0 B/s wr, 13 op/s 2024-03-10T00:52:48.524 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:52:48 smithi170 ceph-mon[27620]: pgmap v8: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 23 KiB/s rd, 0 B/s wr, 13 op/s 2024-03-10T00:52:48.951 INFO:journalctl@ceph.iscsi.iscsi.a.smithi170.stdout:Mar 10 00:52:48 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-iscsi-iscsi-a[50134]: debug there is no tcmu-runner data available 2024-03-10T00:52:49.588 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:52:49 smithi106 ceph-mon[25742]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-10T00:52:49.589 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:52:49 smithi106 ceph-mon[33460]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-10T00:52:49.647 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:52:49 smithi170 ceph-mon[27620]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-10T00:52:49.677 INFO:teuthology.orchestra.run.smithi106.stdout:107374182437 2024-03-10T00:52:49.677 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:831bc188fa69f143bbba69ac76119e784500e8e3 shell --fsid 75c37498-de77-11ee-95c7-87774f69a715 -- ceph osd last-stat-seq osd.3 2024-03-10T00:52:50.515 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:52:50 smithi106 ceph-mon[33460]: pgmap v9: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 20 KiB/s rd, 0 B/s wr, 11 op/s 2024-03-10T00:52:50.515 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:52:50 smithi106 ceph-mon[25742]: pgmap v9: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 20 KiB/s rd, 0 B/s wr, 11 op/s 2024-03-10T00:52:50.541 INFO:teuthology.orchestra.run.smithi106.stdout:163208757275 2024-03-10T00:52:50.541 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:831bc188fa69f143bbba69ac76119e784500e8e3 shell --fsid 75c37498-de77-11ee-95c7-87774f69a715 -- ceph osd last-stat-seq osd.5 2024-03-10T00:52:50.774 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:52:50 smithi170 ceph-mon[27620]: pgmap v9: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 20 KiB/s rd, 0 B/s wr, 11 op/s 2024-03-10T00:52:51.754 INFO:journalctl@ceph.mgr.y.smithi106.stdout:Mar 10 00:52:51 smithi106 ceph-75c37498-de77-11ee-95c7-87774f69a715-mgr-y[25997]: ::ffff:172.21.15.170 - - [10/Mar/2024:00:52:51] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-10T00:52:52.633 INFO:teuthology.orchestra.run.smithi106.stdout:77309411370 2024-03-10T00:52:52.633 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:831bc188fa69f143bbba69ac76119e784500e8e3 shell --fsid 75c37498-de77-11ee-95c7-87774f69a715 -- ceph osd last-stat-seq osd.2 2024-03-10T00:52:52.658 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:52:52 smithi106 ceph-mon[25742]: pgmap v10: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-03-10T00:52:52.658 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:52:52 smithi106 ceph-mon[33460]: pgmap v10: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-03-10T00:52:53.024 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:52:52 smithi170 ceph-mon[27620]: pgmap v10: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-03-10T00:52:53.367 INFO:teuthology.orchestra.run.smithi106.stdout:231928234000 2024-03-10T00:52:53.367 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:831bc188fa69f143bbba69ac76119e784500e8e3 shell --fsid 75c37498-de77-11ee-95c7-87774f69a715 -- ceph osd last-stat-seq osd.7 2024-03-10T00:52:53.928 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:52:53 smithi106 ceph-mon[25742]: from='mgr.14634 172.21.15.106:0/2720500564' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-10T00:52:53.929 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:52:53 smithi106 ceph-mon[33460]: from='mgr.14634 172.21.15.106:0/2720500564' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-10T00:52:54.024 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:52:53 smithi170 ceph-mon[27620]: from='mgr.14634 172.21.15.106:0/2720500564' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-10T00:52:54.633 INFO:teuthology.orchestra.run.smithi106.stdout:55834574897 2024-03-10T00:52:54.634 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:831bc188fa69f143bbba69ac76119e784500e8e3 shell --fsid 75c37498-de77-11ee-95c7-87774f69a715 -- ceph osd last-stat-seq osd.1 2024-03-10T00:52:55.024 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:52:54 smithi170 ceph-mon[27620]: pgmap v11: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-03-10T00:52:55.088 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:52:54 smithi106 ceph-mon[33460]: pgmap v11: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-03-10T00:52:55.088 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:52:54 smithi106 ceph-mon[25742]: pgmap v11: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-03-10T00:52:55.195 INFO:teuthology.orchestra.run.smithi106.stdout:34359738422 2024-03-10T00:52:55.195 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:831bc188fa69f143bbba69ac76119e784500e8e3 shell --fsid 75c37498-de77-11ee-95c7-87774f69a715 -- ceph osd last-stat-seq osd.0 2024-03-10T00:52:55.452 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/75c37498-de77-11ee-95c7-87774f69a715/mon.a/config 2024-03-10T00:52:55.469 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/75c37498-de77-11ee-95c7-87774f69a715/mon.a/config 2024-03-10T00:52:55.473 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/75c37498-de77-11ee-95c7-87774f69a715/mon.a/config 2024-03-10T00:52:56.560 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/75c37498-de77-11ee-95c7-87774f69a715/mon.a/config 2024-03-10T00:52:56.561 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/75c37498-de77-11ee-95c7-87774f69a715/mon.a/config 2024-03-10T00:52:56.562 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/75c37498-de77-11ee-95c7-87774f69a715/mon.a/config 2024-03-10T00:52:56.928 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:52:56 smithi106 ceph-mon[33460]: pgmap v12: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-03-10T00:52:56.928 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:52:56 smithi106 ceph-mon[25742]: pgmap v12: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-03-10T00:52:57.024 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:52:56 smithi170 ceph-mon[27620]: pgmap v12: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-03-10T00:52:58.919 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:52:58 smithi106 ceph-mon[33460]: pgmap v13: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-10T00:52:58.919 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:52:58 smithi106 ceph-mon[25742]: pgmap v13: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-10T00:52:59.024 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:52:58 smithi170 ceph-mon[27620]: pgmap v13: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-10T00:52:59.024 INFO:journalctl@ceph.iscsi.iscsi.a.smithi170.stdout:Mar 10 00:52:58 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-iscsi-iscsi-a[50134]: debug there is no tcmu-runner data available 2024-03-10T00:52:59.838 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:52:59 smithi106 ceph-mon[25742]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-10T00:52:59.838 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:52:59 smithi106 ceph-mon[33460]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-10T00:53:00.024 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:52:59 smithi170 ceph-mon[27620]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-10T00:53:00.838 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:53:00 smithi106 ceph-mon[25742]: pgmap v14: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-10T00:53:00.838 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:53:00 smithi106 ceph-mon[33460]: pgmap v14: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-10T00:53:01.024 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:53:00 smithi170 ceph-mon[27620]: pgmap v14: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-10T00:53:01.521 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/75c37498-de77-11ee-95c7-87774f69a715/mon.a/config 2024-03-10T00:53:01.522 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/75c37498-de77-11ee-95c7-87774f69a715/mon.a/config 2024-03-10T00:53:01.838 INFO:journalctl@ceph.mgr.y.smithi106.stdout:Mar 10 00:53:01 smithi106 ceph-75c37498-de77-11ee-95c7-87774f69a715-mgr-y[25997]: ::ffff:172.21.15.170 - - [10/Mar/2024:00:53:01] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-10T00:53:02.796 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:53:02 smithi106 ceph-mon[33460]: pgmap v15: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-10T00:53:02.796 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:53:02 smithi106 ceph-mon[25742]: pgmap v15: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-10T00:53:03.024 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:53:02 smithi170 ceph-mon[27620]: pgmap v15: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-10T00:53:04.825 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:53:04 smithi106 ceph-mon[25742]: pgmap v16: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-10T00:53:04.825 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:53:04 smithi106 ceph-mon[33460]: pgmap v16: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-10T00:53:04.893 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:53:04 smithi170 ceph-mon[27620]: pgmap v16: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-10T00:53:05.442 INFO:teuthology.orchestra.run.smithi106.stdout:128849018914 2024-03-10T00:53:05.774 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:53:05 smithi170 ceph-mon[27620]: from='client.? 172.21.15.106:0/1212948067' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-03-10T00:53:05.838 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:53:05 smithi106 ceph-mon[25742]: from='client.? 172.21.15.106:0/1212948067' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-03-10T00:53:05.838 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:53:05 smithi106 ceph-mon[33460]: from='client.? 172.21.15.106:0/1212948067' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-03-10T00:53:06.838 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:53:06 smithi106 ceph-mon[25742]: pgmap v17: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-10T00:53:06.838 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:53:06 smithi106 ceph-mon[33460]: pgmap v17: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-10T00:53:07.017 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:53:06 smithi170 ceph-mon[27620]: pgmap v17: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-10T00:53:07.838 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:53:07 smithi106 ceph-mon[25742]: from='client.? 172.21.15.106:0/2295288405' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-03-10T00:53:07.838 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:53:07 smithi106 ceph-mon[33460]: from='client.? 172.21.15.106:0/2295288405' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-03-10T00:53:07.880 INFO:teuthology.orchestra.run.smithi106.stdout:197568495640 2024-03-10T00:53:08.024 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:53:07 smithi170 ceph-mon[27620]: from='client.? 172.21.15.106:0/2295288405' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-03-10T00:53:08.582 INFO:tasks.cephadm.ceph_manager.ceph:need seq 197568495636 got 197568495640 for osd.6 2024-03-10T00:53:08.583 DEBUG:teuthology.parallel:result is None 2024-03-10T00:53:08.836 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:53:08 smithi106 ceph-mon[33460]: pgmap v18: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-10T00:53:08.836 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:53:08 smithi106 ceph-mon[33460]: from='mgr.14634 172.21.15.106:0/2720500564' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-10T00:53:08.836 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:53:08 smithi106 ceph-mon[33460]: from='client.? 172.21.15.106:0/1091929970' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-03-10T00:53:08.836 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:53:08 smithi106 ceph-mon[25742]: pgmap v18: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-10T00:53:08.837 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:53:08 smithi106 ceph-mon[25742]: from='mgr.14634 172.21.15.106:0/2720500564' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-10T00:53:08.837 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:53:08 smithi106 ceph-mon[25742]: from='client.? 172.21.15.106:0/1091929970' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-03-10T00:53:09.024 INFO:journalctl@ceph.iscsi.iscsi.a.smithi170.stdout:Mar 10 00:53:08 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-iscsi-iscsi-a[50134]: debug there is no tcmu-runner data available 2024-03-10T00:53:09.024 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:53:08 smithi170 ceph-mon[27620]: pgmap v18: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-10T00:53:09.024 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:53:08 smithi170 ceph-mon[27620]: from='mgr.14634 172.21.15.106:0/2720500564' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-10T00:53:09.025 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:53:08 smithi170 ceph-mon[27620]: from='client.? 172.21.15.106:0/1091929970' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-03-10T00:53:09.437 INFO:teuthology.orchestra.run.smithi106.stdout:107374182441 2024-03-10T00:53:09.838 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:53:09 smithi106 ceph-mon[33460]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-10T00:53:09.838 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:53:09 smithi106 ceph-mon[33460]: from='client.? 172.21.15.106:0/1976532298' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-03-10T00:53:09.839 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:53:09 smithi106 ceph-mon[25742]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-10T00:53:09.839 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:53:09 smithi106 ceph-mon[25742]: from='client.? 172.21.15.106:0/1976532298' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-03-10T00:53:10.012 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:53:09 smithi170 ceph-mon[27620]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-10T00:53:10.012 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:53:09 smithi170 ceph-mon[27620]: from='client.? 172.21.15.106:0/1976532298' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-03-10T00:53:10.672 INFO:tasks.cephadm.ceph_manager.ceph:need seq 128849018911 got 128849018914 for osd.4 2024-03-10T00:53:10.672 DEBUG:teuthology.parallel:result is None 2024-03-10T00:53:10.973 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:53:10 smithi106 ceph-mon[33460]: pgmap v19: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-10T00:53:10.973 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:53:10 smithi106 ceph-mon[25742]: pgmap v19: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-10T00:53:11.024 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:53:10 smithi170 ceph-mon[27620]: pgmap v19: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-10T00:53:11.582 INFO:journalctl@ceph.mgr.y.smithi106.stdout:Mar 10 00:53:11 smithi106 ceph-75c37498-de77-11ee-95c7-87774f69a715-mgr-y[25997]: ::ffff:172.21.15.170 - - [10/Mar/2024:00:53:11] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-10T00:53:11.812 INFO:teuthology.orchestra.run.smithi106.stdout:55834574900 2024-03-10T00:53:11.908 INFO:teuthology.orchestra.run.smithi106.stdout:34359738425 2024-03-10T00:53:12.013 INFO:teuthology.orchestra.run.smithi106.stdout:77309411375 2024-03-10T00:53:12.825 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:53:12 smithi106 ceph-mon[25742]: pgmap v20: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-03-10T00:53:12.825 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:53:12 smithi106 ceph-mon[25742]: from='client.? 172.21.15.106:0/4030414779' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-03-10T00:53:12.825 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:53:12 smithi106 ceph-mon[25742]: from='client.? 172.21.15.106:0/3031524185' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-03-10T00:53:12.825 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:53:12 smithi106 ceph-mon[33460]: pgmap v20: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-03-10T00:53:12.826 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:53:12 smithi106 ceph-mon[33460]: from='client.? 172.21.15.106:0/4030414779' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-03-10T00:53:12.826 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:53:12 smithi106 ceph-mon[33460]: from='client.? 172.21.15.106:0/3031524185' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-03-10T00:53:13.024 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:53:12 smithi170 ceph-mon[27620]: pgmap v20: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-03-10T00:53:13.024 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:53:12 smithi170 ceph-mon[27620]: from='client.? 172.21.15.106:0/4030414779' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-03-10T00:53:13.024 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:53:12 smithi170 ceph-mon[27620]: from='client.? 172.21.15.106:0/3031524185' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-03-10T00:53:13.535 INFO:teuthology.orchestra.run.smithi106.stdout:163208757280 2024-03-10T00:53:13.838 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:53:13 smithi106 ceph-mon[25742]: from='client.? 172.21.15.106:0/1612783262' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-03-10T00:53:13.838 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:53:13 smithi106 ceph-mon[33460]: from='client.? 172.21.15.106:0/1612783262' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-03-10T00:53:13.953 INFO:tasks.cephadm.ceph_manager.ceph:need seq 34359738422 got 34359738425 for osd.0 2024-03-10T00:53:13.954 DEBUG:teuthology.parallel:result is None 2024-03-10T00:53:14.024 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:53:13 smithi170 ceph-mon[27620]: from='client.? 172.21.15.106:0/1612783262' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-03-10T00:53:14.552 INFO:tasks.cephadm.ceph_manager.ceph:need seq 107374182437 got 107374182441 for osd.3 2024-03-10T00:53:14.552 DEBUG:teuthology.parallel:result is None 2024-03-10T00:53:14.784 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:53:14 smithi106 ceph-mon[33460]: pgmap v21: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-03-10T00:53:14.784 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:53:14 smithi106 ceph-mon[25742]: pgmap v21: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-03-10T00:53:15.024 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:53:14 smithi170 ceph-mon[27620]: pgmap v21: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-03-10T00:53:16.338 INFO:journalctl@ceph.alertmanager.a.smithi106.stdout:Mar 10 00:53:15 smithi106 systemd[1]: Starting Ceph alertmanager.a for 75c37498-de77-11ee-95c7-87774f69a715... 2024-03-10T00:53:16.483 INFO:tasks.cephadm.ceph_manager.ceph:need seq 55834574897 got 55834574900 for osd.1 2024-03-10T00:53:16.483 DEBUG:teuthology.parallel:result is None 2024-03-10T00:53:16.500 INFO:tasks.cephadm.ceph_manager.ceph:need seq 163208757275 got 163208757280 for osd.5 2024-03-10T00:53:16.500 DEBUG:teuthology.parallel:result is None 2024-03-10T00:53:16.950 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:53:16 smithi106 ceph-mon[33460]: pgmap v22: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-10T00:53:16.950 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:53:16 smithi106 ceph-mon[25742]: pgmap v22: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-10T00:53:16.980 INFO:tasks.cephadm.ceph_manager.ceph:need seq 77309411370 got 77309411375 for osd.2 2024-03-10T00:53:16.980 DEBUG:teuthology.parallel:result is None 2024-03-10T00:53:17.024 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:53:16 smithi170 ceph-mon[27620]: pgmap v22: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-10T00:53:17.456 INFO:teuthology.orchestra.run.smithi106.stdout:231928234005 2024-03-10T00:53:17.497 INFO:journalctl@ceph.alertmanager.a.smithi106.stdout:Mar 10 00:53:17 smithi106 podman[65691]: 2024-03-10 00:53:17.150146131 +0000 UTC m=+0.096862286 image pull c8568f914cd25b2062c44e9f79f9c18da6e3b85fe0c47a12a2191c61426c2b19 quay.io/prometheus/alertmanager:v0.25.0 2024-03-10T00:53:17.497 INFO:journalctl@ceph.alertmanager.a.smithi106.stdout:Mar 10 00:53:17 smithi106 podman[65691]: 2024-03-10 00:53:17.35419413 +0000 UTC m=+0.300910271 volume create 31471fbefba53045ac0dfe2604d04ad0894fba8896fd801af5dc0230cda0b16d 2024-03-10T00:53:17.497 INFO:journalctl@ceph.alertmanager.a.smithi106.stdout:Mar 10 00:53:17 smithi106 podman[65691]: 2024-03-10 00:53:17.462982036 +0000 UTC m=+0.409698176 container create d5b698eacae7ffd2e524a0b4c908c8a8efc22ce9c1628ee6ea49ffbca227446b (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-75c37498-de77-11ee-95c7-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-03-10T00:53:17.838 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:53:17 smithi106 ceph-mon[25742]: from='client.? 172.21.15.106:0/3385110367' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-03-10T00:53:17.838 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:53:17 smithi106 ceph-mon[33460]: from='client.? 172.21.15.106:0/3385110367' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-03-10T00:53:18.024 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:53:17 smithi170 ceph-mon[27620]: from='client.? 172.21.15.106:0/3385110367' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-03-10T00:53:18.442 INFO:tasks.cephadm.ceph_manager.ceph:need seq 231928234000 got 231928234005 for osd.7 2024-03-10T00:53:18.442 DEBUG:teuthology.parallel:result is None 2024-03-10T00:53:18.442 INFO:tasks.cephadm.ceph_manager.ceph:waiting for clean 2024-03-10T00:53:18.443 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:831bc188fa69f143bbba69ac76119e784500e8e3 shell --fsid 75c37498-de77-11ee-95c7-87774f69a715 -- ceph pg dump --format=json 2024-03-10T00:53:18.674 INFO:journalctl@ceph.alertmanager.a.smithi106.stdout:Mar 10 00:53:18 smithi106 podman[65691]: 2024-03-10 00:53:18.570008782 +0000 UTC m=+1.516724948 container init d5b698eacae7ffd2e524a0b4c908c8a8efc22ce9c1628ee6ea49ffbca227446b (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-75c37498-de77-11ee-95c7-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-03-10T00:53:18.674 INFO:journalctl@ceph.alertmanager.a.smithi106.stdout:Mar 10 00:53:18 smithi106 podman[65691]: 2024-03-10 00:53:18.576014357 +0000 UTC m=+1.522730497 container start d5b698eacae7ffd2e524a0b4c908c8a8efc22ce9c1628ee6ea49ffbca227446b (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-75c37498-de77-11ee-95c7-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-03-10T00:53:18.674 INFO:journalctl@ceph.alertmanager.a.smithi106.stdout:Mar 10 00:53:18 smithi106 ceph-75c37498-de77-11ee-95c7-87774f69a715-alertmanager-a[65783]: ts=2024-03-10T00:53:18.588Z caller=main.go:240 level=info msg="Starting Alertmanager" version="(version=0.25.0, branch=HEAD, revision=258fab7cdd551f2cf251ed0348f0ad7289aee789)" 2024-03-10T00:53:18.674 INFO:journalctl@ceph.alertmanager.a.smithi106.stdout:Mar 10 00:53:18 smithi106 ceph-75c37498-de77-11ee-95c7-87774f69a715-alertmanager-a[65783]: ts=2024-03-10T00:53:18.588Z caller=main.go:241 level=info build_context="(go=go1.19.4, user=root@abe866dd5717, date=20221222-14:51:36)" 2024-03-10T00:53:18.674 INFO:journalctl@ceph.alertmanager.a.smithi106.stdout:Mar 10 00:53:18 smithi106 ceph-75c37498-de77-11ee-95c7-87774f69a715-alertmanager-a[65783]: ts=2024-03-10T00:53:18.589Z caller=cluster.go:185 level=info component=cluster msg="setting advertise address explicitly" addr=172.21.15.106 port=9094 2024-03-10T00:53:18.674 INFO:journalctl@ceph.alertmanager.a.smithi106.stdout:Mar 10 00:53:18 smithi106 ceph-75c37498-de77-11ee-95c7-87774f69a715-alertmanager-a[65783]: ts=2024-03-10T00:53:18.590Z caller=cluster.go:681 level=info component=cluster msg="Waiting for gossip to settle..." interval=2s 2024-03-10T00:53:18.674 INFO:journalctl@ceph.alertmanager.a.smithi106.stdout:Mar 10 00:53:18 smithi106 ceph-75c37498-de77-11ee-95c7-87774f69a715-alertmanager-a[65783]: ts=2024-03-10T00:53:18.625Z caller=coordinator.go:113 level=info component=configuration msg="Loading configuration file" file=/etc/alertmanager/alertmanager.yml 2024-03-10T00:53:18.675 INFO:journalctl@ceph.alertmanager.a.smithi106.stdout:Mar 10 00:53:18 smithi106 ceph-75c37498-de77-11ee-95c7-87774f69a715-alertmanager-a[65783]: ts=2024-03-10T00:53:18.626Z caller=coordinator.go:126 level=info component=configuration msg="Completed loading of configuration file" file=/etc/alertmanager/alertmanager.yml 2024-03-10T00:53:18.675 INFO:journalctl@ceph.alertmanager.a.smithi106.stdout:Mar 10 00:53:18 smithi106 ceph-75c37498-de77-11ee-95c7-87774f69a715-alertmanager-a[65783]: ts=2024-03-10T00:53:18.629Z caller=tls_config.go:232 level=info msg="Listening on" address=[::]:9093 2024-03-10T00:53:18.675 INFO:journalctl@ceph.alertmanager.a.smithi106.stdout:Mar 10 00:53:18 smithi106 ceph-75c37498-de77-11ee-95c7-87774f69a715-alertmanager-a[65783]: ts=2024-03-10T00:53:18.629Z caller=tls_config.go:235 level=info msg="TLS is disabled." http2=false address=[::]:9093 2024-03-10T00:53:18.764 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/75c37498-de77-11ee-95c7-87774f69a715/mon.a/config 2024-03-10T00:53:18.926 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:53:18 smithi106 ceph-mon[25742]: pgmap v23: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-03-10T00:53:18.927 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:53:18 smithi106 ceph-mon[33460]: pgmap v23: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-03-10T00:53:18.927 INFO:journalctl@ceph.alertmanager.a.smithi106.stdout:Mar 10 00:53:18 smithi106 bash[65691]: d5b698eacae7ffd2e524a0b4c908c8a8efc22ce9c1628ee6ea49ffbca227446b 2024-03-10T00:53:18.927 INFO:journalctl@ceph.alertmanager.a.smithi106.stdout:Mar 10 00:53:18 smithi106 systemd[1]: Started Ceph alertmanager.a for 75c37498-de77-11ee-95c7-87774f69a715. 2024-03-10T00:53:19.024 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:53:18 smithi170 ceph-mon[27620]: pgmap v23: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-03-10T00:53:19.024 INFO:journalctl@ceph.iscsi.iscsi.a.smithi170.stdout:Mar 10 00:53:18 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-iscsi-iscsi-a[50134]: debug there is no tcmu-runner data available 2024-03-10T00:53:19.245 INFO:journalctl@ceph.mgr.y.smithi106.stdout:Mar 10 00:53:18 smithi106 ceph-75c37498-de77-11ee-95c7-87774f69a715-mgr-y[25997]: [10/Mar/2024:00:53:18] ENGINE Bus STOPPING 2024-03-10T00:53:19.246 INFO:journalctl@ceph.mgr.y.smithi106.stdout:Mar 10 00:53:18 smithi106 ceph-75c37498-de77-11ee-95c7-87774f69a715-mgr-y[25997]: [10/Mar/2024:00:53:18] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-03-10T00:53:19.246 INFO:journalctl@ceph.mgr.y.smithi106.stdout:Mar 10 00:53:18 smithi106 ceph-75c37498-de77-11ee-95c7-87774f69a715-mgr-y[25997]: [10/Mar/2024:00:53:18] ENGINE Bus STOPPED 2024-03-10T00:53:19.246 INFO:journalctl@ceph.mgr.y.smithi106.stdout:Mar 10 00:53:18 smithi106 ceph-75c37498-de77-11ee-95c7-87774f69a715-mgr-y[25997]: [10/Mar/2024:00:53:18] ENGINE Bus STARTING 2024-03-10T00:53:19.246 INFO:journalctl@ceph.mgr.y.smithi106.stdout:Mar 10 00:53:19 smithi106 ceph-75c37498-de77-11ee-95c7-87774f69a715-mgr-y[25997]: [10/Mar/2024:00:53:19] ENGINE Serving on http://:::9283 2024-03-10T00:53:19.246 INFO:journalctl@ceph.mgr.y.smithi106.stdout:Mar 10 00:53:19 smithi106 ceph-75c37498-de77-11ee-95c7-87774f69a715-mgr-y[25997]: [10/Mar/2024:00:53:19] ENGINE Bus STARTED 2024-03-10T00:53:19.838 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:53:19 smithi106 ceph-mon[33460]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-10T00:53:19.838 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:53:19 smithi106 ceph-mon[33460]: from='mgr.14634 172.21.15.106:0/2720500564' entity='mgr.y' 2024-03-10T00:53:19.838 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:53:19 smithi106 ceph-mon[33460]: from='mgr.14634 172.21.15.106:0/2720500564' entity='mgr.y' 2024-03-10T00:53:19.838 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:53:19 smithi106 ceph-mon[33460]: from='mgr.14634 172.21.15.106:0/2720500564' entity='mgr.y' 2024-03-10T00:53:19.838 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:53:19 smithi106 ceph-mon[33460]: from='mgr.14634 172.21.15.106:0/2720500564' entity='mgr.y' 2024-03-10T00:53:19.839 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:53:19 smithi106 ceph-mon[33460]: Regenerating cephadm self-signed grafana TLS certificates 2024-03-10T00:53:19.839 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:53:19 smithi106 ceph-mon[33460]: from='mgr.14634 172.21.15.106:0/2720500564' entity='mgr.y' 2024-03-10T00:53:19.839 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:53:19 smithi106 ceph-mon[33460]: from='mgr.14634 172.21.15.106:0/2720500564' entity='mgr.y' 2024-03-10T00:53:19.839 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:53:19 smithi106 ceph-mon[33460]: from='mgr.14634 172.21.15.106:0/2720500564' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-03-10T00:53:19.839 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:53:19 smithi106 ceph-mon[33460]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-03-10T00:53:19.839 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:53:19 smithi106 ceph-mon[33460]: from='mgr.14634 172.21.15.106:0/2720500564' entity='mgr.y' 2024-03-10T00:53:19.839 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:53:19 smithi106 ceph-mon[33460]: Deploying daemon grafana.a on smithi170 2024-03-10T00:53:19.839 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:53:19 smithi106 ceph-mon[25742]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-10T00:53:19.839 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:53:19 smithi106 ceph-mon[25742]: from='mgr.14634 172.21.15.106:0/2720500564' entity='mgr.y' 2024-03-10T00:53:19.839 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:53:19 smithi106 ceph-mon[25742]: from='mgr.14634 172.21.15.106:0/2720500564' entity='mgr.y' 2024-03-10T00:53:19.839 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:53:19 smithi106 ceph-mon[25742]: from='mgr.14634 172.21.15.106:0/2720500564' entity='mgr.y' 2024-03-10T00:53:19.839 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:53:19 smithi106 ceph-mon[25742]: from='mgr.14634 172.21.15.106:0/2720500564' entity='mgr.y' 2024-03-10T00:53:19.840 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:53:19 smithi106 ceph-mon[25742]: Regenerating cephadm self-signed grafana TLS certificates 2024-03-10T00:53:19.840 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:53:19 smithi106 ceph-mon[25742]: from='mgr.14634 172.21.15.106:0/2720500564' entity='mgr.y' 2024-03-10T00:53:19.840 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:53:19 smithi106 ceph-mon[25742]: from='mgr.14634 172.21.15.106:0/2720500564' entity='mgr.y' 2024-03-10T00:53:19.840 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:53:19 smithi106 ceph-mon[25742]: from='mgr.14634 172.21.15.106:0/2720500564' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-03-10T00:53:19.840 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:53:19 smithi106 ceph-mon[25742]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-03-10T00:53:19.840 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:53:19 smithi106 ceph-mon[25742]: from='mgr.14634 172.21.15.106:0/2720500564' entity='mgr.y' 2024-03-10T00:53:19.840 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:53:19 smithi106 ceph-mon[25742]: Deploying daemon grafana.a on smithi170 2024-03-10T00:53:20.024 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:53:19 smithi170 ceph-mon[27620]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-10T00:53:20.024 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:53:19 smithi170 ceph-mon[27620]: from='mgr.14634 172.21.15.106:0/2720500564' entity='mgr.y' 2024-03-10T00:53:20.024 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:53:19 smithi170 ceph-mon[27620]: from='mgr.14634 172.21.15.106:0/2720500564' entity='mgr.y' 2024-03-10T00:53:20.024 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:53:19 smithi170 ceph-mon[27620]: from='mgr.14634 172.21.15.106:0/2720500564' entity='mgr.y' 2024-03-10T00:53:20.025 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:53:19 smithi170 ceph-mon[27620]: from='mgr.14634 172.21.15.106:0/2720500564' entity='mgr.y' 2024-03-10T00:53:20.025 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:53:19 smithi170 ceph-mon[27620]: Regenerating cephadm self-signed grafana TLS certificates 2024-03-10T00:53:20.025 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:53:19 smithi170 ceph-mon[27620]: from='mgr.14634 172.21.15.106:0/2720500564' entity='mgr.y' 2024-03-10T00:53:20.025 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:53:19 smithi170 ceph-mon[27620]: from='mgr.14634 172.21.15.106:0/2720500564' entity='mgr.y' 2024-03-10T00:53:20.025 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:53:19 smithi170 ceph-mon[27620]: from='mgr.14634 172.21.15.106:0/2720500564' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-03-10T00:53:20.025 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:53:19 smithi170 ceph-mon[27620]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-03-10T00:53:20.025 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:53:19 smithi170 ceph-mon[27620]: from='mgr.14634 172.21.15.106:0/2720500564' entity='mgr.y' 2024-03-10T00:53:20.025 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:53:19 smithi170 ceph-mon[27620]: Deploying daemon grafana.a on smithi170 2024-03-10T00:53:20.838 INFO:journalctl@ceph.alertmanager.a.smithi106.stdout:Mar 10 00:53:20 smithi106 ceph-75c37498-de77-11ee-95c7-87774f69a715-alertmanager-a[65783]: ts=2024-03-10T00:53:20.591Z caller=cluster.go:706 level=info component=cluster msg="gossip not settled" polls=0 before=0 now=1 elapsed=2.00003438s 2024-03-10T00:53:21.189 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:53:20 smithi106 ceph-mon[25742]: pgmap v24: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-10T00:53:21.189 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:53:20 smithi106 ceph-mon[33460]: pgmap v24: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-10T00:53:21.274 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:53:20 smithi170 ceph-mon[27620]: pgmap v24: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-10T00:53:21.838 INFO:journalctl@ceph.mgr.y.smithi106.stdout:Mar 10 00:53:21 smithi106 ceph-75c37498-de77-11ee-95c7-87774f69a715-mgr-y[25997]: ::ffff:172.21.15.170 - - [10/Mar/2024:00:53:21] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-10T00:53:22.281 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-03-10T00:53:22.283 INFO:teuthology.orchestra.run.smithi106.stderr:dumped all 2024-03-10T00:53:23.274 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:53:22 smithi170 ceph-mon[27620]: pgmap v25: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-10T00:53:23.274 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:53:22 smithi170 ceph-mon[27620]: from='client.14766 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-10T00:53:23.274 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:53:22 smithi170 ceph-mon[27620]: from='mgr.14634 172.21.15.106:0/2720500564' entity='mgr.y' 2024-03-10T00:53:23.274 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:53:22 smithi170 ceph-mon[27620]: from='mgr.14634 172.21.15.106:0/2720500564' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-10T00:53:23.338 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:53:22 smithi106 ceph-mon[25742]: pgmap v25: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-10T00:53:23.339 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:53:22 smithi106 ceph-mon[25742]: from='client.14766 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-10T00:53:23.339 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:53:22 smithi106 ceph-mon[25742]: from='mgr.14634 172.21.15.106:0/2720500564' entity='mgr.y' 2024-03-10T00:53:23.339 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:53:22 smithi106 ceph-mon[25742]: from='mgr.14634 172.21.15.106:0/2720500564' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-10T00:53:23.339 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:53:22 smithi106 ceph-mon[33460]: pgmap v25: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-10T00:53:23.339 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:53:22 smithi106 ceph-mon[33460]: from='client.14766 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-10T00:53:23.339 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:53:22 smithi106 ceph-mon[33460]: from='mgr.14634 172.21.15.106:0/2720500564' entity='mgr.y' 2024-03-10T00:53:23.339 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:53:22 smithi106 ceph-mon[33460]: from='mgr.14634 172.21.15.106:0/2720500564' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-10T00:53:23.371 INFO:teuthology.orchestra.run.smithi106.stdout:{"pg_ready":true,"pg_map":{"version":25,"stamp":"2024-03-10T00:53:21.570960+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":596318,"num_objects":230,"num_object_clones":0,"num_object_copies":690,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":230,"num_whiteouts":0,"num_read":1347,"num_read_kb":1158,"num_write":873,"num_write_kb":4027,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":882,"ondisk_log_size":882,"up":396,"acting":396,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":396,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749928448,"kb_used":244760,"kb_used_data":7804,"kb_used_omap":12,"kb_used_meta":236851,"kb_avail":749683688,"statfs":{"total":767926730752,"available":767676096512,"internally_reserved":0,"allocated":7991296,"data_stored":4313786,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":242535984},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003082"},"pg_stats":[{"pgid":"6.1a","version":"0'0","reported_seq":17,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-10T00:52:38.020535+0000","last_change":"2024-03-10T00:51:59.144683+0000","last_active":"2024-03-10T00:52:38.020535+0000","last_peered":"2024-03-10T00:52:38.020535+0000","last_clean":"2024-03-10T00:52:38.020535+0000","last_became_active":"2024-03-10T00:51:59.144339+0000","last_became_peered":"2024-03-10T00:51:59.144339+0000","last_unstale":"2024-03-10T00:52:38.020535+0000","last_undegraded":"2024-03-10T00:52:38.020535+0000","last_fullsized":"2024-03-10T00:52:38.020535+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-10T00:51:58.121476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-10T00:51:58.121476+0000","last_clean_scrub_stamp":"2024-03-10T00:51:58.121476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T10:25:17.519277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":21,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-10T00:52:37.551023+0000","last_change":"2024-03-10T00:51:57.136590+0000","last_active":"2024-03-10T00:52:37.551023+0000","last_peered":"2024-03-10T00:52:37.551023+0000","last_clean":"2024-03-10T00:52:37.551023+0000","last_became_active":"2024-03-10T00:51:57.136304+0000","last_became_peered":"2024-03-10T00:51:57.136304+0000","last_unstale":"2024-03-10T00:52:37.551023+0000","last_undegraded":"2024-03-10T00:52:37.551023+0000","last_fullsized":"2024-03-10T00:52:37.551023+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-10T00:51:56.116216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-10T00:51:56.116216+0000","last_clean_scrub_stamp":"2024-03-10T00:51:56.116216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T11:23:56.466435+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":29,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-10T00:52:38.020997+0000","last_change":"2024-03-10T00:51:53.110073+0000","last_active":"2024-03-10T00:52:38.020997+0000","last_peered":"2024-03-10T00:52:38.020997+0000","last_clean":"2024-03-10T00:52:38.020997+0000","last_became_active":"2024-03-10T00:51:53.109784+0000","last_became_peered":"2024-03-10T00:51:53.109784+0000","last_unstale":"2024-03-10T00:52:38.020997+0000","last_undegraded":"2024-03-10T00:52:38.020997+0000","last_fullsized":"2024-03-10T00:52:38.020997+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-10T00:51:52.085771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-10T00:51:52.085771+0000","last_clean_scrub_stamp":"2024-03-10T00:51:52.085771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T12:15:50.790714+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"67'15","reported_seq":53,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-10T00:52:38.019958+0000","last_change":"2024-03-10T00:51:55.122281+0000","last_active":"2024-03-10T00:52:38.019958+0000","last_peered":"2024-03-10T00:52:38.019958+0000","last_clean":"2024-03-10T00:52:38.019958+0000","last_became_active":"2024-03-10T00:51:55.121983+0000","last_became_peered":"2024-03-10T00:51:55.121983+0000","last_unstale":"2024-03-10T00:52:38.019958+0000","last_undegraded":"2024-03-10T00:52:38.019958+0000","last_fullsized":"2024-03-10T00:52:38.019958+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-10T00:51:54.096929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-10T00:51:54.096929+0000","last_clean_scrub_stamp":"2024-03-10T00:51:54.096929+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T09:37:56.832827+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"67'1","reported_seq":18,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-10T00:52:38.021355+0000","last_change":"2024-03-10T00:51:59.266237+0000","last_active":"2024-03-10T00:52:38.021355+0000","last_peered":"2024-03-10T00:52:38.021355+0000","last_clean":"2024-03-10T00:52:38.021355+0000","last_became_active":"2024-03-10T00:51:59.265827+0000","last_became_peered":"2024-03-10T00:51:59.265827+0000","last_unstale":"2024-03-10T00:52:38.021355+0000","last_undegraded":"2024-03-10T00:52:38.021355+0000","last_fullsized":"2024-03-10T00:52:38.021355+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-10T00:51:58.121476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-10T00:51:58.121476+0000","last_clean_scrub_stamp":"2024-03-10T00:51:58.121476+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T08:53:28.833221+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":21,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-10T00:52:38.020092+0000","last_change":"2024-03-10T00:51:57.132759+0000","last_active":"2024-03-10T00:52:38.020092+0000","last_peered":"2024-03-10T00:52:38.020092+0000","last_clean":"2024-03-10T00:52:38.020092+0000","last_became_active":"2024-03-10T00:51:57.132450+0000","last_became_peered":"2024-03-10T00:51:57.132450+0000","last_unstale":"2024-03-10T00:52:38.020092+0000","last_undegraded":"2024-03-10T00:52:38.020092+0000","last_fullsized":"2024-03-10T00:52:38.020092+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-10T00:51:56.116216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-10T00:51:56.116216+0000","last_clean_scrub_stamp":"2024-03-10T00:51:56.116216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T10:05:25.513309+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":29,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-10T00:52:38.022580+0000","last_change":"2024-03-10T00:51:53.111428+0000","last_active":"2024-03-10T00:52:38.022580+0000","last_peered":"2024-03-10T00:52:38.022580+0000","last_clean":"2024-03-10T00:52:38.022580+0000","last_became_active":"2024-03-10T00:51:53.108723+0000","last_became_peered":"2024-03-10T00:51:53.108723+0000","last_unstale":"2024-03-10T00:52:38.022580+0000","last_undegraded":"2024-03-10T00:52:38.022580+0000","last_fullsized":"2024-03-10T00:52:38.022580+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-10T00:51:52.085771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-10T00:51:52.085771+0000","last_clean_scrub_stamp":"2024-03-10T00:51:52.085771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T01:12:53.162235+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"67'41","reported_seq":100,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-10T00:52:38.020948+0000","last_change":"2024-03-10T00:51:55.115459+0000","last_active":"2024-03-10T00:52:38.020948+0000","last_peered":"2024-03-10T00:52:38.020948+0000","last_clean":"2024-03-10T00:52:38.020948+0000","last_became_active":"2024-03-10T00:51:55.115184+0000","last_became_peered":"2024-03-10T00:51:55.115184+0000","last_unstale":"2024-03-10T00:52:38.020948+0000","last_undegraded":"2024-03-10T00:52:38.020948+0000","last_fullsized":"2024-03-10T00:52:38.020948+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-10T00:51:54.096929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-10T00:51:54.096929+0000","last_clean_scrub_stamp":"2024-03-10T00:51:54.096929+0000","objects_scrubbed":0,"log_size":41,"log_dups_size":0,"ondisk_log_size":41,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T05:39:59.836491+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":62,"num_read_kb":49,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":17,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-10T00:52:38.020011+0000","last_change":"2024-03-10T00:51:59.144757+0000","last_active":"2024-03-10T00:52:38.020011+0000","last_peered":"2024-03-10T00:52:38.020011+0000","last_clean":"2024-03-10T00:52:38.020011+0000","last_became_active":"2024-03-10T00:51:59.144453+0000","last_became_peered":"2024-03-10T00:51:59.144453+0000","last_unstale":"2024-03-10T00:52:38.020011+0000","last_undegraded":"2024-03-10T00:52:38.020011+0000","last_fullsized":"2024-03-10T00:52:38.020011+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-10T00:51:58.121476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-10T00:51:58.121476+0000","last_clean_scrub_stamp":"2024-03-10T00:51:58.121476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T05:08:50.523240+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":21,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-10T00:52:37.552210+0000","last_change":"2024-03-10T00:51:57.137078+0000","last_active":"2024-03-10T00:52:37.552210+0000","last_peered":"2024-03-10T00:52:37.552210+0000","last_clean":"2024-03-10T00:52:37.552210+0000","last_became_active":"2024-03-10T00:51:57.136723+0000","last_became_peered":"2024-03-10T00:51:57.136723+0000","last_unstale":"2024-03-10T00:52:37.552210+0000","last_undegraded":"2024-03-10T00:52:37.552210+0000","last_fullsized":"2024-03-10T00:52:37.552210+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-10T00:51:56.116216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-10T00:51:56.116216+0000","last_clean_scrub_stamp":"2024-03-10T00:51:56.116216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T07:12:56.594081+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":29,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-10T00:52:37.553141+0000","last_change":"2024-03-10T00:51:53.114545+0000","last_active":"2024-03-10T00:52:37.553141+0000","last_peered":"2024-03-10T00:52:37.553141+0000","last_clean":"2024-03-10T00:52:37.553141+0000","last_became_active":"2024-03-10T00:51:53.114231+0000","last_became_peered":"2024-03-10T00:51:53.114231+0000","last_unstale":"2024-03-10T00:52:37.553141+0000","last_undegraded":"2024-03-10T00:52:37.553141+0000","last_fullsized":"2024-03-10T00:52:37.553141+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-10T00:51:52.085771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-10T00:51:52.085771+0000","last_clean_scrub_stamp":"2024-03-10T00:51:52.085771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T10:13:15.469773+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"67'21","reported_seq":65,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-10T00:52:38.020683+0000","last_change":"2024-03-10T00:51:55.113106+0000","last_active":"2024-03-10T00:52:38.020683+0000","last_peered":"2024-03-10T00:52:38.020683+0000","last_clean":"2024-03-10T00:52:38.020683+0000","last_became_active":"2024-03-10T00:51:55.112804+0000","last_became_peered":"2024-03-10T00:51:55.112804+0000","last_unstale":"2024-03-10T00:52:38.020683+0000","last_undegraded":"2024-03-10T00:52:38.020683+0000","last_fullsized":"2024-03-10T00:52:38.020683+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-10T00:51:54.096929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-10T00:51:54.096929+0000","last_clean_scrub_stamp":"2024-03-10T00:51:54.096929+0000","objects_scrubbed":0,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T01:08:00.360628+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":35,"num_read_kb":27,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":17,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-10T00:52:37.551907+0000","last_change":"2024-03-10T00:51:59.141885+0000","last_active":"2024-03-10T00:52:37.551907+0000","last_peered":"2024-03-10T00:52:37.551907+0000","last_clean":"2024-03-10T00:52:37.551907+0000","last_became_active":"2024-03-10T00:51:59.141598+0000","last_became_peered":"2024-03-10T00:51:59.141598+0000","last_unstale":"2024-03-10T00:52:37.551907+0000","last_undegraded":"2024-03-10T00:52:37.551907+0000","last_fullsized":"2024-03-10T00:52:37.551907+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-10T00:51:58.121476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-10T00:51:58.121476+0000","last_clean_scrub_stamp":"2024-03-10T00:51:58.121476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T08:31:42.089353+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":21,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-10T00:52:37.662324+0000","last_change":"2024-03-10T00:51:57.139310+0000","last_active":"2024-03-10T00:52:37.662324+0000","last_peered":"2024-03-10T00:52:37.662324+0000","last_clean":"2024-03-10T00:52:37.662324+0000","last_became_active":"2024-03-10T00:51:57.139029+0000","last_became_peered":"2024-03-10T00:51:57.139029+0000","last_unstale":"2024-03-10T00:52:37.662324+0000","last_undegraded":"2024-03-10T00:52:37.662324+0000","last_fullsized":"2024-03-10T00:52:37.662324+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-10T00:51:56.116216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-10T00:51:56.116216+0000","last_clean_scrub_stamp":"2024-03-10T00:51:56.116216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T10:37:53.823563+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":29,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-10T00:52:37.552390+0000","last_change":"2024-03-10T00:51:53.113829+0000","last_active":"2024-03-10T00:52:37.552390+0000","last_peered":"2024-03-10T00:52:37.552390+0000","last_clean":"2024-03-10T00:52:37.552390+0000","last_became_active":"2024-03-10T00:51:53.113552+0000","last_became_peered":"2024-03-10T00:51:53.113552+0000","last_unstale":"2024-03-10T00:52:37.552390+0000","last_undegraded":"2024-03-10T00:52:37.552390+0000","last_fullsized":"2024-03-10T00:52:37.552390+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-10T00:51:52.085771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-10T00:51:52.085771+0000","last_clean_scrub_stamp":"2024-03-10T00:51:52.085771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T01:51:27.567647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"67'11","reported_seq":47,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-10T00:52:38.020908+0000","last_change":"2024-03-10T00:51:55.120009+0000","last_active":"2024-03-10T00:52:38.020908+0000","last_peered":"2024-03-10T00:52:38.020908+0000","last_clean":"2024-03-10T00:52:38.020908+0000","last_became_active":"2024-03-10T00:51:55.119702+0000","last_became_peered":"2024-03-10T00:51:55.119702+0000","last_unstale":"2024-03-10T00:52:38.020908+0000","last_undegraded":"2024-03-10T00:52:38.020908+0000","last_fullsized":"2024-03-10T00:52:38.020908+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-10T00:51:54.096929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-10T00:51:54.096929+0000","last_clean_scrub_stamp":"2024-03-10T00:51:54.096929+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T06:14:04.512628+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":20,"num_read_kb":15,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":17,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-10T00:52:38.021320+0000","last_change":"2024-03-10T00:51:59.263679+0000","last_active":"2024-03-10T00:52:38.021320+0000","last_peered":"2024-03-10T00:52:38.021320+0000","last_clean":"2024-03-10T00:52:38.021320+0000","last_became_active":"2024-03-10T00:51:59.263348+0000","last_became_peered":"2024-03-10T00:51:59.263348+0000","last_unstale":"2024-03-10T00:52:38.021320+0000","last_undegraded":"2024-03-10T00:52:38.021320+0000","last_fullsized":"2024-03-10T00:52:38.021320+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-10T00:51:58.121476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-10T00:51:58.121476+0000","last_clean_scrub_stamp":"2024-03-10T00:51:58.121476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T07:09:44.346803+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":21,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-10T00:52:37.552017+0000","last_change":"2024-03-10T00:51:57.135032+0000","last_active":"2024-03-10T00:52:37.552017+0000","last_peered":"2024-03-10T00:52:37.552017+0000","last_clean":"2024-03-10T00:52:37.552017+0000","last_became_active":"2024-03-10T00:51:57.134727+0000","last_became_peered":"2024-03-10T00:51:57.134727+0000","last_unstale":"2024-03-10T00:52:37.552017+0000","last_undegraded":"2024-03-10T00:52:37.552017+0000","last_fullsized":"2024-03-10T00:52:37.552017+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-10T00:51:56.116216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-10T00:51:56.116216+0000","last_clean_scrub_stamp":"2024-03-10T00:51:56.116216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T11:32:03.708915+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":29,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-10T00:52:38.022041+0000","last_change":"2024-03-10T00:51:53.114924+0000","last_active":"2024-03-10T00:52:38.022041+0000","last_peered":"2024-03-10T00:52:38.022041+0000","last_clean":"2024-03-10T00:52:38.022041+0000","last_became_active":"2024-03-10T00:51:53.114633+0000","last_became_peered":"2024-03-10T00:51:53.114633+0000","last_unstale":"2024-03-10T00:52:38.022041+0000","last_undegraded":"2024-03-10T00:52:38.022041+0000","last_fullsized":"2024-03-10T00:52:38.022041+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-10T00:51:52.085771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-10T00:51:52.085771+0000","last_clean_scrub_stamp":"2024-03-10T00:51:52.085771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T11:34:53.070630+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"67'15","reported_seq":50,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-10T00:52:37.555249+0000","last_change":"2024-03-10T00:51:55.118445+0000","last_active":"2024-03-10T00:52:37.555249+0000","last_peered":"2024-03-10T00:52:37.555249+0000","last_clean":"2024-03-10T00:52:37.555249+0000","last_became_active":"2024-03-10T00:51:55.118151+0000","last_became_peered":"2024-03-10T00:51:55.118151+0000","last_unstale":"2024-03-10T00:52:37.555249+0000","last_undegraded":"2024-03-10T00:52:37.555249+0000","last_fullsized":"2024-03-10T00:52:37.555249+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-10T00:51:54.096929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-10T00:51:54.096929+0000","last_clean_scrub_stamp":"2024-03-10T00:51:54.096929+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T05:13:26.527752+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":17,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-10T00:52:38.022700+0000","last_change":"2024-03-10T00:51:59.263815+0000","last_active":"2024-03-10T00:52:38.022700+0000","last_peered":"2024-03-10T00:52:38.022700+0000","last_clean":"2024-03-10T00:52:38.022700+0000","last_became_active":"2024-03-10T00:51:59.263469+0000","last_became_peered":"2024-03-10T00:51:59.263469+0000","last_unstale":"2024-03-10T00:52:38.022700+0000","last_undegraded":"2024-03-10T00:52:38.022700+0000","last_fullsized":"2024-03-10T00:52:38.022700+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-10T00:51:58.121476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-10T00:51:58.121476+0000","last_clean_scrub_stamp":"2024-03-10T00:51:58.121476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T08:33:25.589194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":21,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-10T00:52:38.020920+0000","last_change":"2024-03-10T00:51:57.135717+0000","last_active":"2024-03-10T00:52:38.020920+0000","last_peered":"2024-03-10T00:52:38.020920+0000","last_clean":"2024-03-10T00:52:38.020920+0000","last_became_active":"2024-03-10T00:51:57.135284+0000","last_became_peered":"2024-03-10T00:51:57.135284+0000","last_unstale":"2024-03-10T00:52:38.020920+0000","last_undegraded":"2024-03-10T00:52:38.020920+0000","last_fullsized":"2024-03-10T00:52:38.020920+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-10T00:51:56.116216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-10T00:51:56.116216+0000","last_clean_scrub_stamp":"2024-03-10T00:51:56.116216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T01:53:23.572680+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":29,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-10T00:52:38.020708+0000","last_change":"2024-03-10T00:51:53.106328+0000","last_active":"2024-03-10T00:52:38.020708+0000","last_peered":"2024-03-10T00:52:38.020708+0000","last_clean":"2024-03-10T00:52:38.020708+0000","last_became_active":"2024-03-10T00:51:53.105960+0000","last_became_peered":"2024-03-10T00:51:53.105960+0000","last_unstale":"2024-03-10T00:52:38.020708+0000","last_undegraded":"2024-03-10T00:52:38.020708+0000","last_fullsized":"2024-03-10T00:52:38.020708+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-10T00:51:52.085771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-10T00:51:52.085771+0000","last_clean_scrub_stamp":"2024-03-10T00:51:52.085771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T08:41:09.650739+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"67'18","reported_seq":60,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-10T00:52:38.022465+0000","last_change":"2024-03-10T00:51:55.121368+0000","last_active":"2024-03-10T00:52:38.022465+0000","last_peered":"2024-03-10T00:52:38.022465+0000","last_clean":"2024-03-10T00:52:38.022465+0000","last_became_active":"2024-03-10T00:51:55.120985+0000","last_became_peered":"2024-03-10T00:51:55.120985+0000","last_unstale":"2024-03-10T00:52:38.022465+0000","last_undegraded":"2024-03-10T00:52:38.022465+0000","last_fullsized":"2024-03-10T00:52:38.022465+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-10T00:51:54.096929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-10T00:51:54.096929+0000","last_clean_scrub_stamp":"2024-03-10T00:51:54.096929+0000","objects_scrubbed":0,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T11:17:42.946770+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":34,"num_read_kb":24,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1c","version":"67'1","reported_seq":19,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-10T00:52:37.662548+0000","last_change":"2024-03-10T00:51:59.135745+0000","last_active":"2024-03-10T00:52:37.662548+0000","last_peered":"2024-03-10T00:52:37.662548+0000","last_clean":"2024-03-10T00:52:37.662548+0000","last_became_active":"2024-03-10T00:51:59.135464+0000","last_became_peered":"2024-03-10T00:51:59.135464+0000","last_unstale":"2024-03-10T00:52:37.662548+0000","last_undegraded":"2024-03-10T00:52:37.662548+0000","last_fullsized":"2024-03-10T00:52:37.662548+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-10T00:51:58.121476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-10T00:51:58.121476+0000","last_clean_scrub_stamp":"2024-03-10T00:51:58.121476+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T03:10:24.567775+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":370,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1f","version":"67'11","reported_seq":53,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-10T00:53:13.173709+0000","last_change":"2024-03-10T00:51:57.137284+0000","last_active":"2024-03-10T00:53:13.173709+0000","last_peered":"2024-03-10T00:53:13.173709+0000","last_clean":"2024-03-10T00:53:13.173709+0000","last_became_active":"2024-03-10T00:51:57.136953+0000","last_became_peered":"2024-03-10T00:51:57.136953+0000","last_unstale":"2024-03-10T00:53:13.173709+0000","last_undegraded":"2024-03-10T00:53:13.173709+0000","last_fullsized":"2024-03-10T00:53:13.173709+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-10T00:51:56.116216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-10T00:51:56.116216+0000","last_clean_scrub_stamp":"2024-03-10T00:51:56.116216+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T07:56:27.964307+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.19","version":"60'1","reported_seq":30,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-10T00:52:37.551149+0000","last_change":"2024-03-10T00:51:53.110576+0000","last_active":"2024-03-10T00:52:37.551149+0000","last_peered":"2024-03-10T00:52:37.551149+0000","last_clean":"2024-03-10T00:52:37.551149+0000","last_became_active":"2024-03-10T00:51:53.110183+0000","last_became_peered":"2024-03-10T00:51:53.110183+0000","last_unstale":"2024-03-10T00:52:37.551149+0000","last_undegraded":"2024-03-10T00:52:37.551149+0000","last_fullsized":"2024-03-10T00:52:37.551149+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-10T00:51:52.085771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-10T00:51:52.085771+0000","last_clean_scrub_stamp":"2024-03-10T00:51:52.085771+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T09:07:47.930342+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"67'16","reported_seq":52,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-10T00:52:38.021602+0000","last_change":"2024-03-10T00:51:55.114063+0000","last_active":"2024-03-10T00:52:38.021602+0000","last_peered":"2024-03-10T00:52:38.021602+0000","last_clean":"2024-03-10T00:52:38.021602+0000","last_became_active":"2024-03-10T00:51:55.113695+0000","last_became_peered":"2024-03-10T00:51:55.113695+0000","last_unstale":"2024-03-10T00:52:38.021602+0000","last_undegraded":"2024-03-10T00:52:38.021602+0000","last_fullsized":"2024-03-10T00:52:38.021602+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-10T00:51:54.096929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-10T00:51:54.096929+0000","last_clean_scrub_stamp":"2024-03-10T00:51:54.096929+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T04:41:10.790916+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":18,"num_write":16,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":17,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-10T00:52:37.552578+0000","last_change":"2024-03-10T00:51:59.140420+0000","last_active":"2024-03-10T00:52:37.552578+0000","last_peered":"2024-03-10T00:52:37.552578+0000","last_clean":"2024-03-10T00:52:37.552578+0000","last_became_active":"2024-03-10T00:51:59.140081+0000","last_became_peered":"2024-03-10T00:51:59.140081+0000","last_unstale":"2024-03-10T00:52:37.552578+0000","last_undegraded":"2024-03-10T00:52:37.552578+0000","last_fullsized":"2024-03-10T00:52:37.552578+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-10T00:51:58.121476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-10T00:51:58.121476+0000","last_clean_scrub_stamp":"2024-03-10T00:51:58.121476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T09:57:19.697840+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":21,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-10T00:52:38.021902+0000","last_change":"2024-03-10T00:51:57.134314+0000","last_active":"2024-03-10T00:52:38.021902+0000","last_peered":"2024-03-10T00:52:38.021902+0000","last_clean":"2024-03-10T00:52:38.021902+0000","last_became_active":"2024-03-10T00:51:57.133999+0000","last_became_peered":"2024-03-10T00:51:57.133999+0000","last_unstale":"2024-03-10T00:52:38.021902+0000","last_undegraded":"2024-03-10T00:52:38.021902+0000","last_fullsized":"2024-03-10T00:52:38.021902+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-10T00:51:56.116216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-10T00:51:56.116216+0000","last_clean_scrub_stamp":"2024-03-10T00:51:56.116216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T11:32:16.620975+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":29,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-10T00:52:38.020186+0000","last_change":"2024-03-10T00:51:53.114064+0000","last_active":"2024-03-10T00:52:38.020186+0000","last_peered":"2024-03-10T00:52:38.020186+0000","last_clean":"2024-03-10T00:52:38.020186+0000","last_became_active":"2024-03-10T00:51:53.113309+0000","last_became_peered":"2024-03-10T00:51:53.113309+0000","last_unstale":"2024-03-10T00:52:38.020186+0000","last_undegraded":"2024-03-10T00:52:38.020186+0000","last_fullsized":"2024-03-10T00:52:38.020186+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-10T00:51:52.085771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-10T00:51:52.085771+0000","last_clean_scrub_stamp":"2024-03-10T00:51:52.085771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T01:44:52.360919+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"67'17","reported_seq":56,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-10T00:52:38.020609+0000","last_change":"2024-03-10T00:51:55.119788+0000","last_active":"2024-03-10T00:52:38.020609+0000","last_peered":"2024-03-10T00:52:38.020609+0000","last_clean":"2024-03-10T00:52:38.020609+0000","last_became_active":"2024-03-10T00:51:55.119563+0000","last_became_peered":"2024-03-10T00:51:55.119563+0000","last_unstale":"2024-03-10T00:52:38.020609+0000","last_undegraded":"2024-03-10T00:52:38.020609+0000","last_fullsized":"2024-03-10T00:52:38.020609+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-10T00:51:54.096929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-10T00:51:54.096929+0000","last_clean_scrub_stamp":"2024-03-10T00:51:54.096929+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T08:26:59.664526+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":29,"num_read_kb":21,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":17,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-10T00:52:37.662319+0000","last_change":"2024-03-10T00:51:59.134997+0000","last_active":"2024-03-10T00:52:37.662319+0000","last_peered":"2024-03-10T00:52:37.662319+0000","last_clean":"2024-03-10T00:52:37.662319+0000","last_became_active":"2024-03-10T00:51:59.134743+0000","last_became_peered":"2024-03-10T00:51:59.134743+0000","last_unstale":"2024-03-10T00:52:37.662319+0000","last_undegraded":"2024-03-10T00:52:37.662319+0000","last_fullsized":"2024-03-10T00:52:37.662319+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-10T00:51:58.121476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-10T00:51:58.121476+0000","last_clean_scrub_stamp":"2024-03-10T00:51:58.121476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T06:56:54.895354+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":21,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-10T00:52:38.019907+0000","last_change":"2024-03-10T00:51:57.139157+0000","last_active":"2024-03-10T00:52:38.019907+0000","last_peered":"2024-03-10T00:52:38.019907+0000","last_clean":"2024-03-10T00:52:38.019907+0000","last_became_active":"2024-03-10T00:51:57.138839+0000","last_became_peered":"2024-03-10T00:51:57.138839+0000","last_unstale":"2024-03-10T00:52:38.019907+0000","last_undegraded":"2024-03-10T00:52:38.019907+0000","last_fullsized":"2024-03-10T00:52:38.019907+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-10T00:51:56.116216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-10T00:51:56.116216+0000","last_clean_scrub_stamp":"2024-03-10T00:51:56.116216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T10:35:01.904822+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":29,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-10T00:52:38.020918+0000","last_change":"2024-03-10T00:51:53.109936+0000","last_active":"2024-03-10T00:52:38.020918+0000","last_peered":"2024-03-10T00:52:38.020918+0000","last_clean":"2024-03-10T00:52:38.020918+0000","last_became_active":"2024-03-10T00:51:53.109660+0000","last_became_peered":"2024-03-10T00:51:53.109660+0000","last_unstale":"2024-03-10T00:52:38.020918+0000","last_undegraded":"2024-03-10T00:52:38.020918+0000","last_fullsized":"2024-03-10T00:52:38.020918+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-10T00:51:52.085771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-10T00:51:52.085771+0000","last_clean_scrub_stamp":"2024-03-10T00:51:52.085771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T02:55:56.270302+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"67'4","reported_seq":31,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-10T00:52:38.020287+0000","last_change":"2024-03-10T00:51:55.120579+0000","last_active":"2024-03-10T00:52:38.020287+0000","last_peered":"2024-03-10T00:52:38.020287+0000","last_clean":"2024-03-10T00:52:38.020287+0000","last_became_active":"2024-03-10T00:51:55.120301+0000","last_became_peered":"2024-03-10T00:51:55.120301+0000","last_unstale":"2024-03-10T00:52:38.020287+0000","last_undegraded":"2024-03-10T00:52:38.020287+0000","last_fullsized":"2024-03-10T00:52:38.020287+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-10T00:51:54.096929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-10T00:51:54.096929+0000","last_clean_scrub_stamp":"2024-03-10T00:51:54.096929+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T01:56:09.559040+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":17,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-10T00:52:38.021162+0000","last_change":"2024-03-10T00:51:59.263814+0000","last_active":"2024-03-10T00:52:38.021162+0000","last_peered":"2024-03-10T00:52:38.021162+0000","last_clean":"2024-03-10T00:52:38.021162+0000","last_became_active":"2024-03-10T00:51:59.263443+0000","last_became_peered":"2024-03-10T00:51:59.263443+0000","last_unstale":"2024-03-10T00:52:38.021162+0000","last_undegraded":"2024-03-10T00:52:38.021162+0000","last_fullsized":"2024-03-10T00:52:38.021162+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-10T00:51:58.121476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-10T00:51:58.121476+0000","last_clean_scrub_stamp":"2024-03-10T00:51:58.121476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T07:11:49.745278+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":21,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-10T00:52:37.662516+0000","last_change":"2024-03-10T00:51:57.138684+0000","last_active":"2024-03-10T00:52:37.662516+0000","last_peered":"2024-03-10T00:52:37.662516+0000","last_clean":"2024-03-10T00:52:37.662516+0000","last_became_active":"2024-03-10T00:51:57.138398+0000","last_became_peered":"2024-03-10T00:51:57.138398+0000","last_unstale":"2024-03-10T00:52:37.662516+0000","last_undegraded":"2024-03-10T00:52:37.662516+0000","last_fullsized":"2024-03-10T00:52:37.662516+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-10T00:51:56.116216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-10T00:51:56.116216+0000","last_clean_scrub_stamp":"2024-03-10T00:51:56.116216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T01:54:53.914173+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":29,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-10T00:52:37.552499+0000","last_change":"2024-03-10T00:51:53.113570+0000","last_active":"2024-03-10T00:52:37.552499+0000","last_peered":"2024-03-10T00:52:37.552499+0000","last_clean":"2024-03-10T00:52:37.552499+0000","last_became_active":"2024-03-10T00:51:53.113228+0000","last_became_peered":"2024-03-10T00:51:53.113228+0000","last_unstale":"2024-03-10T00:52:37.552499+0000","last_undegraded":"2024-03-10T00:52:37.552499+0000","last_fullsized":"2024-03-10T00:52:37.552499+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-10T00:51:52.085771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-10T00:51:52.085771+0000","last_clean_scrub_stamp":"2024-03-10T00:51:52.085771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T08:39:17.810891+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.11","version":"67'17","reported_seq":56,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-10T00:52:38.020801+0000","last_change":"2024-03-10T00:51:55.117051+0000","last_active":"2024-03-10T00:52:38.020801+0000","last_peered":"2024-03-10T00:52:38.020801+0000","last_clean":"2024-03-10T00:52:38.020801+0000","last_became_active":"2024-03-10T00:51:55.116157+0000","last_became_peered":"2024-03-10T00:51:55.116157+0000","last_unstale":"2024-03-10T00:52:38.020801+0000","last_undegraded":"2024-03-10T00:52:38.020801+0000","last_fullsized":"2024-03-10T00:52:38.020801+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-10T00:51:54.096929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-10T00:51:54.096929+0000","last_clean_scrub_stamp":"2024-03-10T00:51:54.096929+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T06:27:53.033220+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":29,"num_read_kb":21,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":17,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-10T00:52:38.019769+0000","last_change":"2024-03-10T00:51:59.147493+0000","last_active":"2024-03-10T00:52:38.019769+0000","last_peered":"2024-03-10T00:52:38.019769+0000","last_clean":"2024-03-10T00:52:38.019769+0000","last_became_active":"2024-03-10T00:51:59.146635+0000","last_became_peered":"2024-03-10T00:51:59.146635+0000","last_unstale":"2024-03-10T00:52:38.019769+0000","last_undegraded":"2024-03-10T00:52:38.019769+0000","last_fullsized":"2024-03-10T00:52:38.019769+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-10T00:51:58.121476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-10T00:51:58.121476+0000","last_clean_scrub_stamp":"2024-03-10T00:51:58.121476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T10:13:21.568724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":21,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-10T00:52:38.021462+0000","last_change":"2024-03-10T00:51:57.137386+0000","last_active":"2024-03-10T00:52:38.021462+0000","last_peered":"2024-03-10T00:52:38.021462+0000","last_clean":"2024-03-10T00:52:38.021462+0000","last_became_active":"2024-03-10T00:51:57.137037+0000","last_became_peered":"2024-03-10T00:51:57.137037+0000","last_unstale":"2024-03-10T00:52:38.021462+0000","last_undegraded":"2024-03-10T00:52:38.021462+0000","last_fullsized":"2024-03-10T00:52:38.021462+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-10T00:51:56.116216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-10T00:51:56.116216+0000","last_clean_scrub_stamp":"2024-03-10T00:51:56.116216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T07:10:20.789695+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":29,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-10T00:52:37.662529+0000","last_change":"2024-03-10T00:51:53.118133+0000","last_active":"2024-03-10T00:52:37.662529+0000","last_peered":"2024-03-10T00:52:37.662529+0000","last_clean":"2024-03-10T00:52:37.662529+0000","last_became_active":"2024-03-10T00:51:53.117852+0000","last_became_peered":"2024-03-10T00:51:53.117852+0000","last_unstale":"2024-03-10T00:52:37.662529+0000","last_undegraded":"2024-03-10T00:52:37.662529+0000","last_fullsized":"2024-03-10T00:52:37.662529+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-10T00:51:52.085771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-10T00:51:52.085771+0000","last_clean_scrub_stamp":"2024-03-10T00:51:52.085771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T12:29:02.349185+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.12","version":"67'9","reported_seq":41,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-10T00:52:37.551868+0000","last_change":"2024-03-10T00:51:55.119656+0000","last_active":"2024-03-10T00:52:37.551868+0000","last_peered":"2024-03-10T00:52:37.551868+0000","last_clean":"2024-03-10T00:52:37.551868+0000","last_became_active":"2024-03-10T00:51:55.119386+0000","last_became_peered":"2024-03-10T00:51:55.119386+0000","last_unstale":"2024-03-10T00:52:37.551868+0000","last_undegraded":"2024-03-10T00:52:37.551868+0000","last_fullsized":"2024-03-10T00:52:37.551868+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-10T00:51:54.096929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-10T00:51:54.096929+0000","last_clean_scrub_stamp":"2024-03-10T00:51:54.096929+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T03:36:40.047496+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":17,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-10T00:52:38.020131+0000","last_change":"2024-03-10T00:51:59.143347+0000","last_active":"2024-03-10T00:52:38.020131+0000","last_peered":"2024-03-10T00:52:38.020131+0000","last_clean":"2024-03-10T00:52:38.020131+0000","last_became_active":"2024-03-10T00:51:59.143014+0000","last_became_peered":"2024-03-10T00:51:59.143014+0000","last_unstale":"2024-03-10T00:52:38.020131+0000","last_undegraded":"2024-03-10T00:52:38.020131+0000","last_fullsized":"2024-03-10T00:52:38.020131+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-10T00:51:58.121476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-10T00:51:58.121476+0000","last_clean_scrub_stamp":"2024-03-10T00:51:58.121476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T08:22:30.688743+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":21,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-10T00:52:37.551632+0000","last_change":"2024-03-10T00:51:57.138450+0000","last_active":"2024-03-10T00:52:37.551632+0000","last_peered":"2024-03-10T00:52:37.551632+0000","last_clean":"2024-03-10T00:52:37.551632+0000","last_became_active":"2024-03-10T00:51:57.138081+0000","last_became_peered":"2024-03-10T00:51:57.138081+0000","last_unstale":"2024-03-10T00:52:37.551632+0000","last_undegraded":"2024-03-10T00:52:37.551632+0000","last_fullsized":"2024-03-10T00:52:37.551632+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-10T00:51:56.116216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-10T00:51:56.116216+0000","last_clean_scrub_stamp":"2024-03-10T00:51:56.116216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T08:53:57.773539+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":29,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-10T00:52:38.021257+0000","last_change":"2024-03-10T00:51:53.113221+0000","last_active":"2024-03-10T00:52:38.021257+0000","last_peered":"2024-03-10T00:52:38.021257+0000","last_clean":"2024-03-10T00:52:38.021257+0000","last_became_active":"2024-03-10T00:51:53.112943+0000","last_became_peered":"2024-03-10T00:51:53.112943+0000","last_unstale":"2024-03-10T00:52:38.021257+0000","last_undegraded":"2024-03-10T00:52:38.021257+0000","last_fullsized":"2024-03-10T00:52:38.021257+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-10T00:51:52.085771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-10T00:51:52.085771+0000","last_clean_scrub_stamp":"2024-03-10T00:51:52.085771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T03:13:22.694962+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"67'11","reported_seq":44,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-10T00:52:38.021027+0000","last_change":"2024-03-10T00:51:55.120437+0000","last_active":"2024-03-10T00:52:38.021027+0000","last_peered":"2024-03-10T00:52:38.021027+0000","last_clean":"2024-03-10T00:52:38.021027+0000","last_became_active":"2024-03-10T00:51:55.120154+0000","last_became_peered":"2024-03-10T00:51:55.120154+0000","last_unstale":"2024-03-10T00:52:38.021027+0000","last_undegraded":"2024-03-10T00:52:38.021027+0000","last_fullsized":"2024-03-10T00:52:38.021027+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-10T00:51:54.096929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-10T00:51:54.096929+0000","last_clean_scrub_stamp":"2024-03-10T00:51:54.096929+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T02:29:54.442646+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":17,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-10T00:52:38.021292+0000","last_change":"2024-03-10T00:51:59.147162+0000","last_active":"2024-03-10T00:52:38.021292+0000","last_peered":"2024-03-10T00:52:38.021292+0000","last_clean":"2024-03-10T00:52:38.021292+0000","last_became_active":"2024-03-10T00:51:59.145954+0000","last_became_peered":"2024-03-10T00:51:59.145954+0000","last_unstale":"2024-03-10T00:52:38.021292+0000","last_undegraded":"2024-03-10T00:52:38.021292+0000","last_fullsized":"2024-03-10T00:52:38.021292+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-10T00:51:58.121476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-10T00:51:58.121476+0000","last_clean_scrub_stamp":"2024-03-10T00:51:58.121476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T06:59:05.277266+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.15","version":"67'11","reported_seq":50,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-10T00:53:13.173729+0000","last_change":"2024-03-10T00:51:57.138389+0000","last_active":"2024-03-10T00:53:13.173729+0000","last_peered":"2024-03-10T00:53:13.173729+0000","last_clean":"2024-03-10T00:53:13.173729+0000","last_became_active":"2024-03-10T00:51:57.138045+0000","last_became_peered":"2024-03-10T00:51:57.138045+0000","last_unstale":"2024-03-10T00:53:13.173729+0000","last_undegraded":"2024-03-10T00:53:13.173729+0000","last_fullsized":"2024-03-10T00:53:13.173729+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-10T00:51:56.116216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-10T00:51:56.116216+0000","last_clean_scrub_stamp":"2024-03-10T00:51:56.116216+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T12:27:13.728540+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":29,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-10T00:52:37.662545+0000","last_change":"2024-03-10T00:51:53.115683+0000","last_active":"2024-03-10T00:52:37.662545+0000","last_peered":"2024-03-10T00:52:37.662545+0000","last_clean":"2024-03-10T00:52:37.662545+0000","last_became_active":"2024-03-10T00:51:53.115238+0000","last_became_peered":"2024-03-10T00:51:53.115238+0000","last_unstale":"2024-03-10T00:52:37.662545+0000","last_undegraded":"2024-03-10T00:52:37.662545+0000","last_fullsized":"2024-03-10T00:52:37.662545+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-10T00:51:52.085771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-10T00:51:52.085771+0000","last_clean_scrub_stamp":"2024-03-10T00:51:52.085771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T08:51:49.918037+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"67'22","reported_seq":64,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-10T00:52:38.021966+0000","last_change":"2024-03-10T00:51:55.118486+0000","last_active":"2024-03-10T00:52:38.021966+0000","last_peered":"2024-03-10T00:52:38.021966+0000","last_clean":"2024-03-10T00:52:38.021966+0000","last_became_active":"2024-03-10T00:51:55.118211+0000","last_became_peered":"2024-03-10T00:51:55.118211+0000","last_unstale":"2024-03-10T00:52:38.021966+0000","last_undegraded":"2024-03-10T00:52:38.021966+0000","last_fullsized":"2024-03-10T00:52:38.021966+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-10T00:51:54.096929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-10T00:51:54.096929+0000","last_clean_scrub_stamp":"2024-03-10T00:51:54.096929+0000","objects_scrubbed":0,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T10:36:53.374217+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":33,"num_read_kb":26,"num_write":22,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":17,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-10T00:52:38.021230+0000","last_change":"2024-03-10T00:51:59.135792+0000","last_active":"2024-03-10T00:52:38.021230+0000","last_peered":"2024-03-10T00:52:38.021230+0000","last_clean":"2024-03-10T00:52:38.021230+0000","last_became_active":"2024-03-10T00:51:59.135512+0000","last_became_peered":"2024-03-10T00:51:59.135512+0000","last_unstale":"2024-03-10T00:52:38.021230+0000","last_undegraded":"2024-03-10T00:52:38.021230+0000","last_fullsized":"2024-03-10T00:52:38.021230+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-10T00:51:58.121476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-10T00:51:58.121476+0000","last_clean_scrub_stamp":"2024-03-10T00:51:58.121476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T10:32:36.652502+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.14","version":"67'11","reported_seq":53,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-10T00:53:13.173518+0000","last_change":"2024-03-10T00:51:57.140078+0000","last_active":"2024-03-10T00:53:13.173518+0000","last_peered":"2024-03-10T00:53:13.173518+0000","last_clean":"2024-03-10T00:53:13.173518+0000","last_became_active":"2024-03-10T00:51:57.139372+0000","last_became_peered":"2024-03-10T00:51:57.139372+0000","last_unstale":"2024-03-10T00:53:13.173518+0000","last_undegraded":"2024-03-10T00:53:13.173518+0000","last_fullsized":"2024-03-10T00:53:13.173518+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-10T00:51:56.116216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-10T00:51:56.116216+0000","last_clean_scrub_stamp":"2024-03-10T00:51:56.116216+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T01:38:48.451295+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":29,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-10T00:52:38.020456+0000","last_change":"2024-03-10T00:51:53.109532+0000","last_active":"2024-03-10T00:52:38.020456+0000","last_peered":"2024-03-10T00:52:38.020456+0000","last_clean":"2024-03-10T00:52:38.020456+0000","last_became_active":"2024-03-10T00:51:53.109212+0000","last_became_peered":"2024-03-10T00:51:53.109212+0000","last_unstale":"2024-03-10T00:52:38.020456+0000","last_undegraded":"2024-03-10T00:52:38.020456+0000","last_fullsized":"2024-03-10T00:52:38.020456+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-10T00:51:52.085771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-10T00:51:52.085771+0000","last_clean_scrub_stamp":"2024-03-10T00:51:52.085771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T11:01:37.331766+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"67'15","reported_seq":53,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-10T00:52:37.553488+0000","last_change":"2024-03-10T00:51:55.117721+0000","last_active":"2024-03-10T00:52:37.553488+0000","last_peered":"2024-03-10T00:52:37.553488+0000","last_clean":"2024-03-10T00:52:37.553488+0000","last_became_active":"2024-03-10T00:51:55.117443+0000","last_became_peered":"2024-03-10T00:51:55.117443+0000","last_unstale":"2024-03-10T00:52:37.553488+0000","last_undegraded":"2024-03-10T00:52:37.553488+0000","last_fullsized":"2024-03-10T00:52:37.553488+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-10T00:51:54.096929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-10T00:51:54.096929+0000","last_clean_scrub_stamp":"2024-03-10T00:51:54.096929+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T12:18:05.172528+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":17,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-10T00:52:37.555605+0000","last_change":"2024-03-10T00:51:59.137447+0000","last_active":"2024-03-10T00:52:37.555605+0000","last_peered":"2024-03-10T00:52:37.555605+0000","last_clean":"2024-03-10T00:52:37.555605+0000","last_became_active":"2024-03-10T00:51:59.137127+0000","last_became_peered":"2024-03-10T00:51:59.137127+0000","last_unstale":"2024-03-10T00:52:37.555605+0000","last_undegraded":"2024-03-10T00:52:37.555605+0000","last_fullsized":"2024-03-10T00:52:37.555605+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-10T00:51:58.121476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-10T00:51:58.121476+0000","last_clean_scrub_stamp":"2024-03-10T00:51:58.121476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T10:40:58.913729+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":21,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-10T00:52:38.022908+0000","last_change":"2024-03-10T00:51:57.139514+0000","last_active":"2024-03-10T00:52:38.022908+0000","last_peered":"2024-03-10T00:52:38.022908+0000","last_clean":"2024-03-10T00:52:38.022908+0000","last_became_active":"2024-03-10T00:51:57.139152+0000","last_became_peered":"2024-03-10T00:51:57.139152+0000","last_unstale":"2024-03-10T00:52:38.022908+0000","last_undegraded":"2024-03-10T00:52:38.022908+0000","last_fullsized":"2024-03-10T00:52:38.022908+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-10T00:51:56.116216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-10T00:51:56.116216+0000","last_clean_scrub_stamp":"2024-03-10T00:51:56.116216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T03:30:30.034401+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":29,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-10T00:52:37.662416+0000","last_change":"2024-03-10T00:51:53.114674+0000","last_active":"2024-03-10T00:52:37.662416+0000","last_peered":"2024-03-10T00:52:37.662416+0000","last_clean":"2024-03-10T00:52:37.662416+0000","last_became_active":"2024-03-10T00:51:53.114376+0000","last_became_peered":"2024-03-10T00:51:53.114376+0000","last_unstale":"2024-03-10T00:52:37.662416+0000","last_undegraded":"2024-03-10T00:52:37.662416+0000","last_fullsized":"2024-03-10T00:52:37.662416+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-10T00:51:52.085771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-10T00:51:52.085771+0000","last_clean_scrub_stamp":"2024-03-10T00:51:52.085771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T08:15:55.103009+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"67'21","reported_seq":65,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-10T00:52:38.021720+0000","last_change":"2024-03-10T00:51:55.109287+0000","last_active":"2024-03-10T00:52:38.021720+0000","last_peered":"2024-03-10T00:52:38.021720+0000","last_clean":"2024-03-10T00:52:38.021720+0000","last_became_active":"2024-03-10T00:51:55.108849+0000","last_became_peered":"2024-03-10T00:51:55.108849+0000","last_unstale":"2024-03-10T00:52:38.021720+0000","last_undegraded":"2024-03-10T00:52:38.021720+0000","last_fullsized":"2024-03-10T00:52:38.021720+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-10T00:51:54.096929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-10T00:51:54.096929+0000","last_clean_scrub_stamp":"2024-03-10T00:51:54.096929+0000","objects_scrubbed":0,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T07:36:31.420197+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":35,"num_read_kb":27,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":17,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-10T00:52:37.662521+0000","last_change":"2024-03-10T00:51:59.266438+0000","last_active":"2024-03-10T00:52:37.662521+0000","last_peered":"2024-03-10T00:52:37.662521+0000","last_clean":"2024-03-10T00:52:37.662521+0000","last_became_active":"2024-03-10T00:51:59.266149+0000","last_became_peered":"2024-03-10T00:51:59.266149+0000","last_unstale":"2024-03-10T00:52:37.662521+0000","last_undegraded":"2024-03-10T00:52:37.662521+0000","last_fullsized":"2024-03-10T00:52:37.662521+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-10T00:51:58.121476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-10T00:51:58.121476+0000","last_clean_scrub_stamp":"2024-03-10T00:51:58.121476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T07:55:15.921266+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":21,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-10T00:52:37.552599+0000","last_change":"2024-03-10T00:51:57.137507+0000","last_active":"2024-03-10T00:52:37.552599+0000","last_peered":"2024-03-10T00:52:37.552599+0000","last_clean":"2024-03-10T00:52:37.552599+0000","last_became_active":"2024-03-10T00:51:57.137235+0000","last_became_peered":"2024-03-10T00:51:57.137235+0000","last_unstale":"2024-03-10T00:52:37.552599+0000","last_undegraded":"2024-03-10T00:52:37.552599+0000","last_fullsized":"2024-03-10T00:52:37.552599+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-10T00:51:56.116216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-10T00:51:56.116216+0000","last_clean_scrub_stamp":"2024-03-10T00:51:56.116216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T02:46:05.371162+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"60'1","reported_seq":37,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-10T00:52:38.019805+0000","last_change":"2024-03-10T00:51:53.114271+0000","last_active":"2024-03-10T00:52:38.019805+0000","last_peered":"2024-03-10T00:52:38.019805+0000","last_clean":"2024-03-10T00:52:38.019805+0000","last_became_active":"2024-03-10T00:51:53.113912+0000","last_became_peered":"2024-03-10T00:51:53.113912+0000","last_unstale":"2024-03-10T00:52:38.019805+0000","last_undegraded":"2024-03-10T00:52:38.019805+0000","last_fullsized":"2024-03-10T00:52:38.019805+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-10T00:51:52.085771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-10T00:51:52.085771+0000","last_clean_scrub_stamp":"2024-03-10T00:51:52.085771+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T02:54:02.668765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":398,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"67'12","reported_seq":46,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-10T00:52:38.022828+0000","last_change":"2024-03-10T00:51:55.119659+0000","last_active":"2024-03-10T00:52:38.022828+0000","last_peered":"2024-03-10T00:52:38.022828+0000","last_clean":"2024-03-10T00:52:38.022828+0000","last_became_active":"2024-03-10T00:51:55.119398+0000","last_became_peered":"2024-03-10T00:51:55.119398+0000","last_unstale":"2024-03-10T00:52:38.022828+0000","last_undegraded":"2024-03-10T00:52:38.022828+0000","last_fullsized":"2024-03-10T00:52:38.022828+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-10T00:51:54.096929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-10T00:51:54.096929+0000","last_clean_scrub_stamp":"2024-03-10T00:51:54.096929+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T06:19:43.507953+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":18,"num_read_kb":14,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":17,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-10T00:52:37.552490+0000","last_change":"2024-03-10T00:51:59.263745+0000","last_active":"2024-03-10T00:52:37.552490+0000","last_peered":"2024-03-10T00:52:37.552490+0000","last_clean":"2024-03-10T00:52:37.552490+0000","last_became_active":"2024-03-10T00:51:59.263366+0000","last_became_peered":"2024-03-10T00:51:59.263366+0000","last_unstale":"2024-03-10T00:52:37.552490+0000","last_undegraded":"2024-03-10T00:52:37.552490+0000","last_fullsized":"2024-03-10T00:52:37.552490+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-10T00:51:58.121476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-10T00:51:58.121476+0000","last_clean_scrub_stamp":"2024-03-10T00:51:58.121476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T04:04:34.367832+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"60'2","reported_seq":45,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-10T00:52:37.662894+0000","last_change":"2024-03-10T00:51:53.119511+0000","last_active":"2024-03-10T00:52:37.662894+0000","last_peered":"2024-03-10T00:52:37.662894+0000","last_clean":"2024-03-10T00:52:37.662894+0000","last_became_active":"2024-03-10T00:51:53.119222+0000","last_became_peered":"2024-03-10T00:51:53.119222+0000","last_unstale":"2024-03-10T00:52:37.662894+0000","last_undegraded":"2024-03-10T00:52:37.662894+0000","last_fullsized":"2024-03-10T00:52:37.662894+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-10T00:51:52.085771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-10T00:51:52.085771+0000","last_clean_scrub_stamp":"2024-03-10T00:51:52.085771+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T06:35:44.779038+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":14,"num_read_kb":14,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"67'27","reported_seq":74,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-10T00:52:37.551960+0000","last_change":"2024-03-10T00:51:55.114646+0000","last_active":"2024-03-10T00:52:37.551960+0000","last_peered":"2024-03-10T00:52:37.551960+0000","last_clean":"2024-03-10T00:52:37.551960+0000","last_became_active":"2024-03-10T00:51:55.114342+0000","last_became_peered":"2024-03-10T00:51:55.114342+0000","last_unstale":"2024-03-10T00:52:37.551960+0000","last_undegraded":"2024-03-10T00:52:37.551960+0000","last_fullsized":"2024-03-10T00:52:37.551960+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-10T00:51:54.096929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-10T00:51:54.096929+0000","last_clean_scrub_stamp":"2024-03-10T00:51:54.096929+0000","objects_scrubbed":0,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T11:30:26.324585+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":44,"num_read_kb":33,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"67'11","reported_seq":50,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-10T00:53:13.173711+0000","last_change":"2024-03-10T00:51:57.143030+0000","last_active":"2024-03-10T00:53:13.173711+0000","last_peered":"2024-03-10T00:53:13.173711+0000","last_clean":"2024-03-10T00:53:13.173711+0000","last_became_active":"2024-03-10T00:51:57.142653+0000","last_became_peered":"2024-03-10T00:51:57.142653+0000","last_unstale":"2024-03-10T00:53:13.173711+0000","last_undegraded":"2024-03-10T00:53:13.173711+0000","last_fullsized":"2024-03-10T00:53:13.173711+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-10T00:51:56.116216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-10T00:51:56.116216+0000","last_clean_scrub_stamp":"2024-03-10T00:51:56.116216+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T05:14:26.367968+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":17,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-10T00:52:38.021258+0000","last_change":"2024-03-10T00:51:59.146184+0000","last_active":"2024-03-10T00:52:38.021258+0000","last_peered":"2024-03-10T00:52:38.021258+0000","last_clean":"2024-03-10T00:52:38.021258+0000","last_became_active":"2024-03-10T00:51:59.145644+0000","last_became_peered":"2024-03-10T00:51:59.145644+0000","last_unstale":"2024-03-10T00:52:38.021258+0000","last_undegraded":"2024-03-10T00:52:38.021258+0000","last_fullsized":"2024-03-10T00:52:38.021258+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-10T00:51:58.121476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-10T00:51:58.121476+0000","last_clean_scrub_stamp":"2024-03-10T00:51:58.121476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T00:54:03.007401+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":29,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-10T00:52:37.662716+0000","last_change":"2024-03-10T00:51:53.118823+0000","last_active":"2024-03-10T00:52:37.662716+0000","last_peered":"2024-03-10T00:52:37.662716+0000","last_clean":"2024-03-10T00:52:37.662716+0000","last_became_active":"2024-03-10T00:51:53.118546+0000","last_became_peered":"2024-03-10T00:51:53.118546+0000","last_unstale":"2024-03-10T00:52:37.662716+0000","last_undegraded":"2024-03-10T00:52:37.662716+0000","last_fullsized":"2024-03-10T00:52:37.662716+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-10T00:51:52.085771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-10T00:51:52.085771+0000","last_clean_scrub_stamp":"2024-03-10T00:51:52.085771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T04:51:14.535355+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"67'24","reported_seq":72,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-10T00:52:38.020372+0000","last_change":"2024-03-10T00:51:55.119156+0000","last_active":"2024-03-10T00:52:38.020372+0000","last_peered":"2024-03-10T00:52:38.020372+0000","last_clean":"2024-03-10T00:52:38.020372+0000","last_became_active":"2024-03-10T00:51:55.118871+0000","last_became_peered":"2024-03-10T00:51:55.118871+0000","last_unstale":"2024-03-10T00:52:38.020372+0000","last_undegraded":"2024-03-10T00:52:38.020372+0000","last_fullsized":"2024-03-10T00:52:38.020372+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-10T00:51:54.096929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-10T00:51:54.096929+0000","last_clean_scrub_stamp":"2024-03-10T00:51:54.096929+0000","objects_scrubbed":0,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T06:07:46.272776+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":32,"num_write":26,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":21,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-10T00:52:37.555488+0000","last_change":"2024-03-10T00:51:57.136934+0000","last_active":"2024-03-10T00:52:37.555488+0000","last_peered":"2024-03-10T00:52:37.555488+0000","last_clean":"2024-03-10T00:52:37.555488+0000","last_became_active":"2024-03-10T00:51:57.136579+0000","last_became_peered":"2024-03-10T00:51:57.136579+0000","last_unstale":"2024-03-10T00:52:37.555488+0000","last_undegraded":"2024-03-10T00:52:37.555488+0000","last_fullsized":"2024-03-10T00:52:37.555488+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-10T00:51:56.116216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-10T00:51:56.116216+0000","last_clean_scrub_stamp":"2024-03-10T00:51:56.116216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T07:56:33.320243+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":17,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-10T00:52:37.662180+0000","last_change":"2024-03-10T00:51:59.137262+0000","last_active":"2024-03-10T00:52:37.662180+0000","last_peered":"2024-03-10T00:52:37.662180+0000","last_clean":"2024-03-10T00:52:37.662180+0000","last_became_active":"2024-03-10T00:51:59.136997+0000","last_became_peered":"2024-03-10T00:51:59.136997+0000","last_unstale":"2024-03-10T00:52:37.662180+0000","last_undegraded":"2024-03-10T00:52:37.662180+0000","last_fullsized":"2024-03-10T00:52:37.662180+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-10T00:51:58.121476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-10T00:51:58.121476+0000","last_clean_scrub_stamp":"2024-03-10T00:51:58.121476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T05:23:57.386618+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":21,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-10T00:52:37.554790+0000","last_change":"2024-03-10T00:51:57.135617+0000","last_active":"2024-03-10T00:52:37.554790+0000","last_peered":"2024-03-10T00:52:37.554790+0000","last_clean":"2024-03-10T00:52:37.554790+0000","last_became_active":"2024-03-10T00:51:57.135026+0000","last_became_peered":"2024-03-10T00:51:57.135026+0000","last_unstale":"2024-03-10T00:52:37.554790+0000","last_undegraded":"2024-03-10T00:52:37.554790+0000","last_fullsized":"2024-03-10T00:52:37.554790+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-10T00:51:56.116216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-10T00:51:56.116216+0000","last_clean_scrub_stamp":"2024-03-10T00:51:56.116216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T08:27:10.673504+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":29,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-10T00:52:37.662875+0000","last_change":"2024-03-10T00:51:53.112002+0000","last_active":"2024-03-10T00:52:37.662875+0000","last_peered":"2024-03-10T00:52:37.662875+0000","last_clean":"2024-03-10T00:52:37.662875+0000","last_became_active":"2024-03-10T00:51:53.111707+0000","last_became_peered":"2024-03-10T00:51:53.111707+0000","last_unstale":"2024-03-10T00:52:37.662875+0000","last_undegraded":"2024-03-10T00:52:37.662875+0000","last_fullsized":"2024-03-10T00:52:37.662875+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-10T00:51:52.085771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-10T00:51:52.085771+0000","last_clean_scrub_stamp":"2024-03-10T00:51:52.085771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T11:33:18.020874+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"67'25","reported_seq":68,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-10T00:52:38.020557+0000","last_change":"2024-03-10T00:51:55.120348+0000","last_active":"2024-03-10T00:52:38.020557+0000","last_peered":"2024-03-10T00:52:38.020557+0000","last_clean":"2024-03-10T00:52:38.020557+0000","last_became_active":"2024-03-10T00:51:55.120009+0000","last_became_peered":"2024-03-10T00:51:55.120009+0000","last_unstale":"2024-03-10T00:52:38.020557+0000","last_undegraded":"2024-03-10T00:52:38.020557+0000","last_fullsized":"2024-03-10T00:52:38.020557+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-10T00:51:54.096929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-10T00:51:54.096929+0000","last_clean_scrub_stamp":"2024-03-10T00:51:54.096929+0000","objects_scrubbed":0,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T09:13:02.827451+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":41,"num_read_kb":29,"num_write":26,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":17,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-10T00:52:38.019995+0000","last_change":"2024-03-10T00:51:59.144510+0000","last_active":"2024-03-10T00:52:38.019995+0000","last_peered":"2024-03-10T00:52:38.019995+0000","last_clean":"2024-03-10T00:52:38.019995+0000","last_became_active":"2024-03-10T00:51:59.144023+0000","last_became_peered":"2024-03-10T00:51:59.144023+0000","last_unstale":"2024-03-10T00:52:38.019995+0000","last_undegraded":"2024-03-10T00:52:38.019995+0000","last_fullsized":"2024-03-10T00:52:38.019995+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-10T00:51:58.121476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-10T00:51:58.121476+0000","last_clean_scrub_stamp":"2024-03-10T00:51:58.121476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T01:27:36.953468+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":21,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-10T00:52:37.554826+0000","last_change":"2024-03-10T00:51:57.134250+0000","last_active":"2024-03-10T00:52:37.554826+0000","last_peered":"2024-03-10T00:52:37.554826+0000","last_clean":"2024-03-10T00:52:37.554826+0000","last_became_active":"2024-03-10T00:51:57.133948+0000","last_became_peered":"2024-03-10T00:51:57.133948+0000","last_unstale":"2024-03-10T00:52:37.554826+0000","last_undegraded":"2024-03-10T00:52:37.554826+0000","last_fullsized":"2024-03-10T00:52:37.554826+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-10T00:51:56.116216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-10T00:51:56.116216+0000","last_clean_scrub_stamp":"2024-03-10T00:51:56.116216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T06:34:24.152549+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":29,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-10T00:52:37.552488+0000","last_change":"2024-03-10T00:51:53.118037+0000","last_active":"2024-03-10T00:52:37.552488+0000","last_peered":"2024-03-10T00:52:37.552488+0000","last_clean":"2024-03-10T00:52:37.552488+0000","last_became_active":"2024-03-10T00:51:53.117612+0000","last_became_peered":"2024-03-10T00:51:53.117612+0000","last_unstale":"2024-03-10T00:52:37.552488+0000","last_undegraded":"2024-03-10T00:52:37.552488+0000","last_fullsized":"2024-03-10T00:52:37.552488+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-10T00:51:52.085771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-10T00:51:52.085771+0000","last_clean_scrub_stamp":"2024-03-10T00:51:52.085771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T08:40:54.041208+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"67'15","reported_seq":53,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-10T00:52:38.022247+0000","last_change":"2024-03-10T00:51:55.119392+0000","last_active":"2024-03-10T00:52:38.022247+0000","last_peered":"2024-03-10T00:52:38.022247+0000","last_clean":"2024-03-10T00:52:38.022247+0000","last_became_active":"2024-03-10T00:51:55.119073+0000","last_became_peered":"2024-03-10T00:51:55.119073+0000","last_unstale":"2024-03-10T00:52:38.022247+0000","last_undegraded":"2024-03-10T00:52:38.022247+0000","last_fullsized":"2024-03-10T00:52:38.022247+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-10T00:51:54.096929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-10T00:51:54.096929+0000","last_clean_scrub_stamp":"2024-03-10T00:51:54.096929+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T09:09:22.672882+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":17,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-10T00:52:38.022308+0000","last_change":"2024-03-10T00:51:59.137543+0000","last_active":"2024-03-10T00:52:38.022308+0000","last_peered":"2024-03-10T00:52:38.022308+0000","last_clean":"2024-03-10T00:52:38.022308+0000","last_became_active":"2024-03-10T00:51:59.137259+0000","last_became_peered":"2024-03-10T00:51:59.137259+0000","last_unstale":"2024-03-10T00:52:38.022308+0000","last_undegraded":"2024-03-10T00:52:38.022308+0000","last_fullsized":"2024-03-10T00:52:38.022308+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-10T00:51:58.121476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-10T00:51:58.121476+0000","last_clean_scrub_stamp":"2024-03-10T00:51:58.121476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T04:45:53.316568+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"64'2","reported_seq":38,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-10T00:52:37.552124+0000","last_change":"2024-03-10T00:51:57.132946+0000","last_active":"2024-03-10T00:52:37.552124+0000","last_peered":"2024-03-10T00:52:37.552124+0000","last_clean":"2024-03-10T00:52:37.552124+0000","last_became_active":"2024-03-10T00:51:53.112432+0000","last_became_peered":"2024-03-10T00:51:53.112432+0000","last_unstale":"2024-03-10T00:52:37.552124+0000","last_undegraded":"2024-03-10T00:52:37.552124+0000","last_fullsized":"2024-03-10T00:52:37.552124+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-10T00:51:52.085771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-10T00:51:52.085771+0000","last_clean_scrub_stamp":"2024-03-10T00:51:52.085771+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T10:49:28.839398+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0015561780000000001,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":29,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-10T00:52:38.020741+0000","last_change":"2024-03-10T00:51:53.115959+0000","last_active":"2024-03-10T00:52:38.020741+0000","last_peered":"2024-03-10T00:52:38.020741+0000","last_clean":"2024-03-10T00:52:38.020741+0000","last_became_active":"2024-03-10T00:51:53.115681+0000","last_became_peered":"2024-03-10T00:51:53.115681+0000","last_unstale":"2024-03-10T00:52:38.020741+0000","last_undegraded":"2024-03-10T00:52:38.020741+0000","last_fullsized":"2024-03-10T00:52:38.020741+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-10T00:51:52.085771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-10T00:51:52.085771+0000","last_clean_scrub_stamp":"2024-03-10T00:51:52.085771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T12:48:38.872556+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.4","version":"67'36","reported_seq":106,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-10T00:53:13.173525+0000","last_change":"2024-03-10T00:51:55.123960+0000","last_active":"2024-03-10T00:53:13.173525+0000","last_peered":"2024-03-10T00:53:13.173525+0000","last_clean":"2024-03-10T00:53:13.173525+0000","last_became_active":"2024-03-10T00:51:55.123642+0000","last_became_peered":"2024-03-10T00:51:55.123642+0000","last_unstale":"2024-03-10T00:53:13.173525+0000","last_undegraded":"2024-03-10T00:53:13.173525+0000","last_fullsized":"2024-03-10T00:53:13.173525+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-10T00:51:54.096929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-10T00:51:54.096929+0000","last_clean_scrub_stamp":"2024-03-10T00:51:54.096929+0000","objects_scrubbed":0,"log_size":36,"log_dups_size":0,"ondisk_log_size":36,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T12:12:52.388286+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":60,"num_read_kb":44,"num_write":32,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":21,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-10T00:52:38.020475+0000","last_change":"2024-03-10T00:51:57.136454+0000","last_active":"2024-03-10T00:52:38.020475+0000","last_peered":"2024-03-10T00:52:38.020475+0000","last_clean":"2024-03-10T00:52:38.020475+0000","last_became_active":"2024-03-10T00:51:57.136014+0000","last_became_peered":"2024-03-10T00:51:57.136014+0000","last_unstale":"2024-03-10T00:52:38.020475+0000","last_undegraded":"2024-03-10T00:52:38.020475+0000","last_fullsized":"2024-03-10T00:52:38.020475+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-10T00:51:56.116216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-10T00:51:56.116216+0000","last_clean_scrub_stamp":"2024-03-10T00:51:56.116216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T11:48:07.545530+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":17,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-10T00:52:37.662976+0000","last_change":"2024-03-10T00:51:59.265490+0000","last_active":"2024-03-10T00:52:37.662976+0000","last_peered":"2024-03-10T00:52:37.662976+0000","last_clean":"2024-03-10T00:52:37.662976+0000","last_became_active":"2024-03-10T00:51:59.265155+0000","last_became_peered":"2024-03-10T00:51:59.265155+0000","last_unstale":"2024-03-10T00:52:37.662976+0000","last_undegraded":"2024-03-10T00:52:37.662976+0000","last_fullsized":"2024-03-10T00:52:37.662976+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-10T00:51:58.121476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-10T00:51:58.121476+0000","last_clean_scrub_stamp":"2024-03-10T00:51:58.121476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T08:51:31.161077+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":29,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-10T00:52:37.551063+0000","last_change":"2024-03-10T00:51:53.111526+0000","last_active":"2024-03-10T00:52:37.551063+0000","last_peered":"2024-03-10T00:52:37.551063+0000","last_clean":"2024-03-10T00:52:37.551063+0000","last_became_active":"2024-03-10T00:51:53.111103+0000","last_became_peered":"2024-03-10T00:51:53.111103+0000","last_unstale":"2024-03-10T00:52:37.551063+0000","last_undegraded":"2024-03-10T00:52:37.551063+0000","last_fullsized":"2024-03-10T00:52:37.551063+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-10T00:51:52.085771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-10T00:51:52.085771+0000","last_clean_scrub_stamp":"2024-03-10T00:51:52.085771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T05:26:32.767017+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":36,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-10T00:52:37.555661+0000","last_change":"2024-03-10T00:51:57.124791+0000","last_active":"2024-03-10T00:52:37.555661+0000","last_peered":"2024-03-10T00:52:37.555661+0000","last_clean":"2024-03-10T00:52:37.555661+0000","last_became_active":"2024-03-10T00:51:53.112252+0000","last_became_peered":"2024-03-10T00:51:53.112252+0000","last_unstale":"2024-03-10T00:52:37.555661+0000","last_undegraded":"2024-03-10T00:52:37.555661+0000","last_fullsized":"2024-03-10T00:52:37.555661+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-10T00:51:52.085771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-10T00:51:52.085771+0000","last_clean_scrub_stamp":"2024-03-10T00:51:52.085771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T09:14:05.516707+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00067091600000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"67'19","reported_seq":64,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-10T00:52:37.553069+0000","last_change":"2024-03-10T00:51:55.118163+0000","last_active":"2024-03-10T00:52:37.553069+0000","last_peered":"2024-03-10T00:52:37.553069+0000","last_clean":"2024-03-10T00:52:37.553069+0000","last_became_active":"2024-03-10T00:51:55.117829+0000","last_became_peered":"2024-03-10T00:51:55.117829+0000","last_unstale":"2024-03-10T00:52:37.553069+0000","last_undegraded":"2024-03-10T00:52:37.553069+0000","last_fullsized":"2024-03-10T00:52:37.553069+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-10T00:51:54.096929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-10T00:51:54.096929+0000","last_clean_scrub_stamp":"2024-03-10T00:51:54.096929+0000","objects_scrubbed":0,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T08:45:59.775240+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":39,"num_read_kb":27,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":21,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-10T00:52:37.554923+0000","last_change":"2024-03-10T00:51:57.138045+0000","last_active":"2024-03-10T00:52:37.554923+0000","last_peered":"2024-03-10T00:52:37.554923+0000","last_clean":"2024-03-10T00:52:37.554923+0000","last_became_active":"2024-03-10T00:51:57.136939+0000","last_became_peered":"2024-03-10T00:51:57.136939+0000","last_unstale":"2024-03-10T00:52:37.554923+0000","last_undegraded":"2024-03-10T00:52:37.554923+0000","last_fullsized":"2024-03-10T00:52:37.554923+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-10T00:51:56.116216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-10T00:51:56.116216+0000","last_clean_scrub_stamp":"2024-03-10T00:51:56.116216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T06:34:37.358733+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":17,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-10T00:52:37.552739+0000","last_change":"2024-03-10T00:51:59.141162+0000","last_active":"2024-03-10T00:52:37.552739+0000","last_peered":"2024-03-10T00:52:37.552739+0000","last_clean":"2024-03-10T00:52:37.552739+0000","last_became_active":"2024-03-10T00:51:59.140795+0000","last_became_peered":"2024-03-10T00:51:59.140795+0000","last_unstale":"2024-03-10T00:52:37.552739+0000","last_undegraded":"2024-03-10T00:52:37.552739+0000","last_fullsized":"2024-03-10T00:52:37.552739+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-10T00:51:58.121476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-10T00:51:58.121476+0000","last_clean_scrub_stamp":"2024-03-10T00:51:58.121476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T09:31:17.571248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"69'188","reported_seq":341,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-10T00:52:39.656783+0000","last_change":"2024-03-10T00:51:41.219917+0000","last_active":"2024-03-10T00:52:39.656783+0000","last_peered":"2024-03-10T00:52:39.656783+0000","last_clean":"2024-03-10T00:52:39.656783+0000","last_became_active":"2024-03-10T00:51:41.219507+0000","last_became_peered":"2024-03-10T00:51:41.219507+0000","last_unstale":"2024-03-10T00:52:39.656783+0000","last_undegraded":"2024-03-10T00:52:39.656783+0000","last_fullsized":"2024-03-10T00:52:39.656783+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-10T00:49:25.306169+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-10T00:49:25.306169+0000","last_clean_scrub_stamp":"2024-03-10T00:49:25.306169+0000","objects_scrubbed":0,"log_size":188,"log_dups_size":0,"ondisk_log_size":188,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T12:12:42.766777+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":242,"num_read_kb":333,"num_write":245,"num_write_kb":3982,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":29,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-10T00:52:38.020996+0000","last_change":"2024-03-10T00:51:53.115998+0000","last_active":"2024-03-10T00:52:38.020996+0000","last_peered":"2024-03-10T00:52:38.020996+0000","last_clean":"2024-03-10T00:52:38.020996+0000","last_became_active":"2024-03-10T00:51:53.115668+0000","last_became_peered":"2024-03-10T00:51:53.115668+0000","last_unstale":"2024-03-10T00:52:38.020996+0000","last_undegraded":"2024-03-10T00:52:38.020996+0000","last_fullsized":"2024-03-10T00:52:38.020996+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-10T00:51:52.085771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-10T00:51:52.085771+0000","last_clean_scrub_stamp":"2024-03-10T00:51:52.085771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T09:59:09.605361+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"67'16","reported_seq":65,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-10T00:53:13.173736+0000","last_change":"2024-03-10T00:51:55.115838+0000","last_active":"2024-03-10T00:53:13.173736+0000","last_peered":"2024-03-10T00:53:13.173736+0000","last_clean":"2024-03-10T00:53:13.173736+0000","last_became_active":"2024-03-10T00:51:55.115529+0000","last_became_peered":"2024-03-10T00:51:55.115529+0000","last_unstale":"2024-03-10T00:53:13.173736+0000","last_undegraded":"2024-03-10T00:53:13.173736+0000","last_fullsized":"2024-03-10T00:53:13.173736+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-10T00:51:54.096929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-10T00:51:54.096929+0000","last_clean_scrub_stamp":"2024-03-10T00:51:54.096929+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T12:11:12.704417+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":25,"num_read_kb":15,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":21,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-10T00:52:37.662703+0000","last_change":"2024-03-10T00:51:57.141083+0000","last_active":"2024-03-10T00:52:37.662703+0000","last_peered":"2024-03-10T00:52:37.662703+0000","last_clean":"2024-03-10T00:52:37.662703+0000","last_became_active":"2024-03-10T00:51:57.140761+0000","last_became_peered":"2024-03-10T00:51:57.140761+0000","last_unstale":"2024-03-10T00:52:37.662703+0000","last_undegraded":"2024-03-10T00:52:37.662703+0000","last_fullsized":"2024-03-10T00:52:37.662703+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-10T00:51:56.116216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-10T00:51:56.116216+0000","last_clean_scrub_stamp":"2024-03-10T00:51:56.116216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T04:13:15.481320+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":17,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-10T00:52:37.551542+0000","last_change":"2024-03-10T00:51:59.142614+0000","last_active":"2024-03-10T00:52:37.551542+0000","last_peered":"2024-03-10T00:52:37.551542+0000","last_clean":"2024-03-10T00:52:37.551542+0000","last_became_active":"2024-03-10T00:51:59.142332+0000","last_became_peered":"2024-03-10T00:51:59.142332+0000","last_unstale":"2024-03-10T00:52:37.551542+0000","last_undegraded":"2024-03-10T00:52:37.551542+0000","last_fullsized":"2024-03-10T00:52:37.551542+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-10T00:51:58.121476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-10T00:51:58.121476+0000","last_clean_scrub_stamp":"2024-03-10T00:51:58.121476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T11:14:07.678440+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":29,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-10T00:52:38.021152+0000","last_change":"2024-03-10T00:51:53.117913+0000","last_active":"2024-03-10T00:52:38.021152+0000","last_peered":"2024-03-10T00:52:38.021152+0000","last_clean":"2024-03-10T00:52:38.021152+0000","last_became_active":"2024-03-10T00:51:53.117308+0000","last_became_peered":"2024-03-10T00:51:53.117308+0000","last_unstale":"2024-03-10T00:52:38.021152+0000","last_undegraded":"2024-03-10T00:52:38.021152+0000","last_fullsized":"2024-03-10T00:52:38.021152+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-10T00:51:52.085771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-10T00:51:52.085771+0000","last_clean_scrub_stamp":"2024-03-10T00:51:52.085771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T06:30:43.045812+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"67'5","reported_seq":118,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-10T00:53:16.585038+0000","last_change":"2024-03-10T00:51:57.134275+0000","last_active":"2024-03-10T00:53:16.585038+0000","last_peered":"2024-03-10T00:53:16.585038+0000","last_clean":"2024-03-10T00:53:16.585038+0000","last_became_active":"2024-03-10T00:51:53.111611+0000","last_became_peered":"2024-03-10T00:51:53.111611+0000","last_unstale":"2024-03-10T00:53:16.585038+0000","last_undegraded":"2024-03-10T00:53:16.585038+0000","last_fullsized":"2024-03-10T00:53:16.585038+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-10T00:51:52.085771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-10T00:51:52.085771+0000","last_clean_scrub_stamp":"2024-03-10T00:51:52.085771+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T08:49:42.631320+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00098723699999999997,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":79,"num_read_kb":74,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"67'12","reported_seq":43,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-10T00:52:38.021378+0000","last_change":"2024-03-10T00:51:55.119330+0000","last_active":"2024-03-10T00:52:38.021378+0000","last_peered":"2024-03-10T00:52:38.021378+0000","last_clean":"2024-03-10T00:52:38.021378+0000","last_became_active":"2024-03-10T00:51:55.119047+0000","last_became_peered":"2024-03-10T00:51:55.119047+0000","last_unstale":"2024-03-10T00:52:38.021378+0000","last_undegraded":"2024-03-10T00:52:38.021378+0000","last_fullsized":"2024-03-10T00:52:38.021378+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-10T00:51:54.096929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-10T00:51:54.096929+0000","last_clean_scrub_stamp":"2024-03-10T00:51:54.096929+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T04:52:14.508539+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":18,"num_read_kb":12,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":21,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-10T00:52:37.552933+0000","last_change":"2024-03-10T00:51:57.140671+0000","last_active":"2024-03-10T00:52:37.552933+0000","last_peered":"2024-03-10T00:52:37.552933+0000","last_clean":"2024-03-10T00:52:37.552933+0000","last_became_active":"2024-03-10T00:51:57.140186+0000","last_became_peered":"2024-03-10T00:51:57.140186+0000","last_unstale":"2024-03-10T00:52:37.552933+0000","last_undegraded":"2024-03-10T00:52:37.552933+0000","last_fullsized":"2024-03-10T00:52:37.552933+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-10T00:51:56.116216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-10T00:51:56.116216+0000","last_clean_scrub_stamp":"2024-03-10T00:51:56.116216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T09:29:31.087034+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":17,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-10T00:52:37.551016+0000","last_change":"2024-03-10T00:51:59.264751+0000","last_active":"2024-03-10T00:52:37.551016+0000","last_peered":"2024-03-10T00:52:37.551016+0000","last_clean":"2024-03-10T00:52:37.551016+0000","last_became_active":"2024-03-10T00:51:59.264375+0000","last_became_peered":"2024-03-10T00:51:59.264375+0000","last_unstale":"2024-03-10T00:52:37.551016+0000","last_undegraded":"2024-03-10T00:52:37.551016+0000","last_fullsized":"2024-03-10T00:52:37.551016+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-10T00:51:58.121476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-10T00:51:58.121476+0000","last_clean_scrub_stamp":"2024-03-10T00:51:58.121476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T01:20:42.245941+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":29,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-10T00:52:37.551937+0000","last_change":"2024-03-10T00:51:53.110714+0000","last_active":"2024-03-10T00:52:37.551937+0000","last_peered":"2024-03-10T00:52:37.551937+0000","last_clean":"2024-03-10T00:52:37.551937+0000","last_became_active":"2024-03-10T00:51:53.110240+0000","last_became_peered":"2024-03-10T00:51:53.110240+0000","last_unstale":"2024-03-10T00:52:37.551937+0000","last_undegraded":"2024-03-10T00:52:37.551937+0000","last_fullsized":"2024-03-10T00:52:37.551937+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-10T00:51:52.085771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-10T00:51:52.085771+0000","last_clean_scrub_stamp":"2024-03-10T00:51:52.085771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T09:51:21.505184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.3","version":"67'25","reported_seq":73,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-10T00:52:38.022041+0000","last_change":"2024-03-10T00:51:55.115268+0000","last_active":"2024-03-10T00:52:38.022041+0000","last_peered":"2024-03-10T00:52:38.022041+0000","last_clean":"2024-03-10T00:52:38.022041+0000","last_became_active":"2024-03-10T00:51:55.114979+0000","last_became_peered":"2024-03-10T00:51:55.114979+0000","last_unstale":"2024-03-10T00:52:38.022041+0000","last_undegraded":"2024-03-10T00:52:38.022041+0000","last_fullsized":"2024-03-10T00:52:38.022041+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-10T00:51:54.096929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-10T00:51:54.096929+0000","last_clean_scrub_stamp":"2024-03-10T00:51:54.096929+0000","objects_scrubbed":0,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T12:51:17.750576+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":48,"num_read_kb":33,"num_write":28,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":21,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-10T00:52:38.020715+0000","last_change":"2024-03-10T00:51:57.135661+0000","last_active":"2024-03-10T00:52:38.020715+0000","last_peered":"2024-03-10T00:52:38.020715+0000","last_clean":"2024-03-10T00:52:38.020715+0000","last_became_active":"2024-03-10T00:51:57.135317+0000","last_became_peered":"2024-03-10T00:51:57.135317+0000","last_unstale":"2024-03-10T00:52:38.020715+0000","last_undegraded":"2024-03-10T00:52:38.020715+0000","last_fullsized":"2024-03-10T00:52:38.020715+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-10T00:51:56.116216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-10T00:51:56.116216+0000","last_clean_scrub_stamp":"2024-03-10T00:51:56.116216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T07:04:45.110526+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":17,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-10T00:52:38.019786+0000","last_change":"2024-03-10T00:51:59.147816+0000","last_active":"2024-03-10T00:52:38.019786+0000","last_peered":"2024-03-10T00:52:38.019786+0000","last_clean":"2024-03-10T00:52:38.019786+0000","last_became_active":"2024-03-10T00:51:59.147088+0000","last_became_peered":"2024-03-10T00:51:59.147088+0000","last_unstale":"2024-03-10T00:52:38.019786+0000","last_undegraded":"2024-03-10T00:52:38.019786+0000","last_fullsized":"2024-03-10T00:52:38.019786+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-10T00:51:58.121476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-10T00:51:58.121476+0000","last_clean_scrub_stamp":"2024-03-10T00:51:58.121476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T02:44:35.968610+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":29,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-10T00:52:37.553346+0000","last_change":"2024-03-10T00:51:53.117039+0000","last_active":"2024-03-10T00:52:37.553346+0000","last_peered":"2024-03-10T00:52:37.553346+0000","last_clean":"2024-03-10T00:52:37.553346+0000","last_became_active":"2024-03-10T00:51:53.116259+0000","last_became_peered":"2024-03-10T00:51:53.116259+0000","last_unstale":"2024-03-10T00:52:37.553346+0000","last_undegraded":"2024-03-10T00:52:37.553346+0000","last_fullsized":"2024-03-10T00:52:37.553346+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-10T00:51:52.085771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-10T00:51:52.085771+0000","last_clean_scrub_stamp":"2024-03-10T00:51:52.085771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T06:38:54.512952+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.2","version":"67'16","reported_seq":52,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-10T00:52:37.551724+0000","last_change":"2024-03-10T00:51:55.119021+0000","last_active":"2024-03-10T00:52:37.551724+0000","last_peered":"2024-03-10T00:52:37.551724+0000","last_clean":"2024-03-10T00:52:37.551724+0000","last_became_active":"2024-03-10T00:51:55.118661+0000","last_became_peered":"2024-03-10T00:51:55.118661+0000","last_unstale":"2024-03-10T00:52:37.551724+0000","last_undegraded":"2024-03-10T00:52:37.551724+0000","last_fullsized":"2024-03-10T00:52:37.551724+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-10T00:51:54.096929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-10T00:51:54.096929+0000","last_clean_scrub_stamp":"2024-03-10T00:51:54.096929+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T07:19:05.573264+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":18,"num_write":16,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3","version":"67'11","reported_seq":50,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-10T00:53:13.173527+0000","last_change":"2024-03-10T00:51:57.135495+0000","last_active":"2024-03-10T00:53:13.173527+0000","last_peered":"2024-03-10T00:53:13.173527+0000","last_clean":"2024-03-10T00:53:13.173527+0000","last_became_active":"2024-03-10T00:51:57.135156+0000","last_became_peered":"2024-03-10T00:51:57.135156+0000","last_unstale":"2024-03-10T00:53:13.173527+0000","last_undegraded":"2024-03-10T00:53:13.173527+0000","last_fullsized":"2024-03-10T00:53:13.173527+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-10T00:51:56.116216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-10T00:51:56.116216+0000","last_clean_scrub_stamp":"2024-03-10T00:51:56.116216+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T04:31:48.567337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":17,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-10T00:52:37.662841+0000","last_change":"2024-03-10T00:51:59.265807+0000","last_active":"2024-03-10T00:52:37.662841+0000","last_peered":"2024-03-10T00:52:37.662841+0000","last_clean":"2024-03-10T00:52:37.662841+0000","last_became_active":"2024-03-10T00:51:59.265510+0000","last_became_peered":"2024-03-10T00:51:59.265510+0000","last_unstale":"2024-03-10T00:52:37.662841+0000","last_undegraded":"2024-03-10T00:52:37.662841+0000","last_fullsized":"2024-03-10T00:52:37.662841+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-10T00:51:58.121476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-10T00:51:58.121476+0000","last_clean_scrub_stamp":"2024-03-10T00:51:58.121476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T06:39:25.973402+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.6","version":"60'1","reported_seq":30,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-10T00:52:38.021499+0000","last_change":"2024-03-10T00:51:53.117648+0000","last_active":"2024-03-10T00:52:38.021499+0000","last_peered":"2024-03-10T00:52:38.021499+0000","last_clean":"2024-03-10T00:52:38.021499+0000","last_became_active":"2024-03-10T00:51:53.117331+0000","last_became_peered":"2024-03-10T00:51:53.117331+0000","last_unstale":"2024-03-10T00:52:38.021499+0000","last_undegraded":"2024-03-10T00:52:38.021499+0000","last_fullsized":"2024-03-10T00:52:38.021499+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-10T00:51:52.085771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-10T00:51:52.085771+0000","last_clean_scrub_stamp":"2024-03-10T00:51:52.085771+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T08:08:21.440050+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1","version":"67'20","reported_seq":58,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-10T00:52:38.020245+0000","last_change":"2024-03-10T00:51:55.115843+0000","last_active":"2024-03-10T00:52:38.020245+0000","last_peered":"2024-03-10T00:52:38.020245+0000","last_clean":"2024-03-10T00:52:38.020245+0000","last_became_active":"2024-03-10T00:51:55.115549+0000","last_became_peered":"2024-03-10T00:51:55.115549+0000","last_unstale":"2024-03-10T00:52:38.020245+0000","last_undegraded":"2024-03-10T00:52:38.020245+0000","last_fullsized":"2024-03-10T00:52:38.020245+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-10T00:51:54.096929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-10T00:51:54.096929+0000","last_clean_scrub_stamp":"2024-03-10T00:51:54.096929+0000","objects_scrubbed":0,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T06:34:56.720783+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":30,"num_read_kb":22,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.0","version":"67'11","reported_seq":50,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-10T00:53:13.173743+0000","last_change":"2024-03-10T00:51:57.141295+0000","last_active":"2024-03-10T00:53:13.173743+0000","last_peered":"2024-03-10T00:53:13.173743+0000","last_clean":"2024-03-10T00:53:13.173743+0000","last_became_active":"2024-03-10T00:51:57.140997+0000","last_became_peered":"2024-03-10T00:51:57.140997+0000","last_unstale":"2024-03-10T00:53:13.173743+0000","last_undegraded":"2024-03-10T00:53:13.173743+0000","last_fullsized":"2024-03-10T00:53:13.173743+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-10T00:51:56.116216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-10T00:51:56.116216+0000","last_clean_scrub_stamp":"2024-03-10T00:51:56.116216+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T10:25:44.950288+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":17,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-10T00:52:38.020575+0000","last_change":"2024-03-10T00:51:59.135775+0000","last_active":"2024-03-10T00:52:38.020575+0000","last_peered":"2024-03-10T00:52:38.020575+0000","last_clean":"2024-03-10T00:52:38.020575+0000","last_became_active":"2024-03-10T00:51:59.135492+0000","last_became_peered":"2024-03-10T00:51:59.135492+0000","last_unstale":"2024-03-10T00:52:38.020575+0000","last_undegraded":"2024-03-10T00:52:38.020575+0000","last_fullsized":"2024-03-10T00:52:38.020575+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-10T00:51:58.121476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-10T00:51:58.121476+0000","last_clean_scrub_stamp":"2024-03-10T00:51:58.121476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T08:55:04.887116+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"60'1","reported_seq":37,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-10T00:52:38.022769+0000","last_change":"2024-03-10T00:51:53.112758+0000","last_active":"2024-03-10T00:52:38.022769+0000","last_peered":"2024-03-10T00:52:38.022769+0000","last_clean":"2024-03-10T00:52:38.022769+0000","last_became_active":"2024-03-10T00:51:53.111694+0000","last_became_peered":"2024-03-10T00:51:53.111694+0000","last_unstale":"2024-03-10T00:52:38.022769+0000","last_undegraded":"2024-03-10T00:52:38.022769+0000","last_fullsized":"2024-03-10T00:52:38.022769+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-10T00:51:52.085771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-10T00:51:52.085771+0000","last_clean_scrub_stamp":"2024-03-10T00:51:52.085771+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T10:03:50.587703+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":875,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.0","version":"67'32","reported_seq":83,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-10T00:52:38.020372+0000","last_change":"2024-03-10T00:51:55.116821+0000","last_active":"2024-03-10T00:52:38.020372+0000","last_peered":"2024-03-10T00:52:38.020372+0000","last_clean":"2024-03-10T00:52:38.020372+0000","last_became_active":"2024-03-10T00:51:55.116480+0000","last_became_peered":"2024-03-10T00:51:55.116480+0000","last_unstale":"2024-03-10T00:52:38.020372+0000","last_undegraded":"2024-03-10T00:52:38.020372+0000","last_fullsized":"2024-03-10T00:52:38.020372+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-10T00:51:54.096929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-10T00:51:54.096929+0000","last_clean_scrub_stamp":"2024-03-10T00:51:54.096929+0000","objects_scrubbed":0,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T07:27:59.886118+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":52,"num_read_kb":38,"num_write":32,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":21,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-10T00:52:38.020090+0000","last_change":"2024-03-10T00:51:57.135984+0000","last_active":"2024-03-10T00:52:38.020090+0000","last_peered":"2024-03-10T00:52:38.020090+0000","last_clean":"2024-03-10T00:52:38.020090+0000","last_became_active":"2024-03-10T00:51:57.135386+0000","last_became_peered":"2024-03-10T00:51:57.135386+0000","last_unstale":"2024-03-10T00:52:38.020090+0000","last_undegraded":"2024-03-10T00:52:38.020090+0000","last_fullsized":"2024-03-10T00:52:38.020090+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-10T00:51:56.116216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-10T00:51:56.116216+0000","last_clean_scrub_stamp":"2024-03-10T00:51:56.116216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T11:18:38.081117+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":17,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-10T00:52:37.553023+0000","last_change":"2024-03-10T00:51:59.141683+0000","last_active":"2024-03-10T00:52:37.553023+0000","last_peered":"2024-03-10T00:52:37.553023+0000","last_clean":"2024-03-10T00:52:37.553023+0000","last_became_active":"2024-03-10T00:51:59.141437+0000","last_became_peered":"2024-03-10T00:51:59.141437+0000","last_unstale":"2024-03-10T00:52:37.553023+0000","last_undegraded":"2024-03-10T00:52:37.553023+0000","last_fullsized":"2024-03-10T00:52:37.553023+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-10T00:51:58.121476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-10T00:51:58.121476+0000","last_clean_scrub_stamp":"2024-03-10T00:51:58.121476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T10:12:35.650969+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.e","version":"67'11","reported_seq":50,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-10T00:53:13.173724+0000","last_change":"2024-03-10T00:51:57.135230+0000","last_active":"2024-03-10T00:53:13.173724+0000","last_peered":"2024-03-10T00:53:13.173724+0000","last_clean":"2024-03-10T00:53:13.173724+0000","last_became_active":"2024-03-10T00:51:57.134202+0000","last_became_peered":"2024-03-10T00:51:57.134202+0000","last_unstale":"2024-03-10T00:53:13.173724+0000","last_undegraded":"2024-03-10T00:53:13.173724+0000","last_fullsized":"2024-03-10T00:53:13.173724+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-10T00:51:56.116216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-10T00:51:56.116216+0000","last_clean_scrub_stamp":"2024-03-10T00:51:56.116216+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T05:00:15.821525+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":29,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-10T00:52:38.020277+0000","last_change":"2024-03-10T00:51:53.112469+0000","last_active":"2024-03-10T00:52:38.020277+0000","last_peered":"2024-03-10T00:52:38.020277+0000","last_clean":"2024-03-10T00:52:38.020277+0000","last_became_active":"2024-03-10T00:51:53.108351+0000","last_became_peered":"2024-03-10T00:51:53.108351+0000","last_unstale":"2024-03-10T00:52:38.020277+0000","last_undegraded":"2024-03-10T00:52:38.020277+0000","last_fullsized":"2024-03-10T00:52:38.020277+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-10T00:51:52.085771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-10T00:51:52.085771+0000","last_clean_scrub_stamp":"2024-03-10T00:51:52.085771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T09:25:32.877715+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"67'15","reported_seq":50,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-10T00:52:37.552908+0000","last_change":"2024-03-10T00:51:55.124009+0000","last_active":"2024-03-10T00:52:37.552908+0000","last_peered":"2024-03-10T00:52:37.552908+0000","last_clean":"2024-03-10T00:52:37.552908+0000","last_became_active":"2024-03-10T00:51:55.123632+0000","last_became_peered":"2024-03-10T00:51:55.123632+0000","last_unstale":"2024-03-10T00:52:37.552908+0000","last_undegraded":"2024-03-10T00:52:37.552908+0000","last_fullsized":"2024-03-10T00:52:37.552908+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-10T00:51:54.096929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-10T00:51:54.096929+0000","last_clean_scrub_stamp":"2024-03-10T00:51:54.096929+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T11:48:51.675319+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":17,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-10T00:52:38.021814+0000","last_change":"2024-03-10T00:51:59.263882+0000","last_active":"2024-03-10T00:52:38.021814+0000","last_peered":"2024-03-10T00:52:38.021814+0000","last_clean":"2024-03-10T00:52:38.021814+0000","last_became_active":"2024-03-10T00:51:59.263462+0000","last_became_peered":"2024-03-10T00:51:59.263462+0000","last_unstale":"2024-03-10T00:52:38.021814+0000","last_undegraded":"2024-03-10T00:52:38.021814+0000","last_fullsized":"2024-03-10T00:52:38.021814+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-10T00:51:58.121476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-10T00:51:58.121476+0000","last_clean_scrub_stamp":"2024-03-10T00:51:58.121476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T01:44:14.373567+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":21,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-10T00:52:37.552905+0000","last_change":"2024-03-10T00:51:57.140451+0000","last_active":"2024-03-10T00:52:37.552905+0000","last_peered":"2024-03-10T00:52:37.552905+0000","last_clean":"2024-03-10T00:52:37.552905+0000","last_became_active":"2024-03-10T00:51:57.139816+0000","last_became_peered":"2024-03-10T00:51:57.139816+0000","last_unstale":"2024-03-10T00:52:37.552905+0000","last_undegraded":"2024-03-10T00:52:37.552905+0000","last_fullsized":"2024-03-10T00:52:37.552905+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-10T00:51:56.116216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-10T00:51:56.116216+0000","last_clean_scrub_stamp":"2024-03-10T00:51:56.116216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T03:29:46.383843+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":29,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-10T00:52:38.020147+0000","last_change":"2024-03-10T00:51:53.107401+0000","last_active":"2024-03-10T00:52:38.020147+0000","last_peered":"2024-03-10T00:52:38.020147+0000","last_clean":"2024-03-10T00:52:38.020147+0000","last_became_active":"2024-03-10T00:51:53.107121+0000","last_became_peered":"2024-03-10T00:51:53.107121+0000","last_unstale":"2024-03-10T00:52:38.020147+0000","last_undegraded":"2024-03-10T00:52:38.020147+0000","last_fullsized":"2024-03-10T00:52:38.020147+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-10T00:51:52.085771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-10T00:51:52.085771+0000","last_clean_scrub_stamp":"2024-03-10T00:51:52.085771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T03:19:09.033740+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"67'17","reported_seq":56,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-10T00:52:38.021182+0000","last_change":"2024-03-10T00:51:55.119294+0000","last_active":"2024-03-10T00:52:38.021182+0000","last_peered":"2024-03-10T00:52:38.021182+0000","last_clean":"2024-03-10T00:52:38.021182+0000","last_became_active":"2024-03-10T00:51:55.118985+0000","last_became_peered":"2024-03-10T00:51:55.118985+0000","last_unstale":"2024-03-10T00:52:38.021182+0000","last_undegraded":"2024-03-10T00:52:38.021182+0000","last_fullsized":"2024-03-10T00:52:38.021182+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-10T00:51:54.096929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-10T00:51:54.096929+0000","last_clean_scrub_stamp":"2024-03-10T00:51:54.096929+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T04:41:40.524309+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":29,"num_read_kb":21,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":17,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-10T00:52:37.554923+0000","last_change":"2024-03-10T00:51:59.139953+0000","last_active":"2024-03-10T00:52:37.554923+0000","last_peered":"2024-03-10T00:52:37.554923+0000","last_clean":"2024-03-10T00:52:37.554923+0000","last_became_active":"2024-03-10T00:51:59.139571+0000","last_became_peered":"2024-03-10T00:51:59.139571+0000","last_unstale":"2024-03-10T00:52:37.554923+0000","last_undegraded":"2024-03-10T00:52:37.554923+0000","last_fullsized":"2024-03-10T00:52:37.554923+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-10T00:51:58.121476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-10T00:51:58.121476+0000","last_clean_scrub_stamp":"2024-03-10T00:51:58.121476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T05:59:57.853376+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":21,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-10T00:52:37.551970+0000","last_change":"2024-03-10T00:51:57.135459+0000","last_active":"2024-03-10T00:52:37.551970+0000","last_peered":"2024-03-10T00:52:37.551970+0000","last_clean":"2024-03-10T00:52:37.551970+0000","last_became_active":"2024-03-10T00:51:57.134429+0000","last_became_peered":"2024-03-10T00:51:57.134429+0000","last_unstale":"2024-03-10T00:52:37.551970+0000","last_undegraded":"2024-03-10T00:52:37.551970+0000","last_fullsized":"2024-03-10T00:52:37.551970+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-10T00:51:56.116216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-10T00:51:56.116216+0000","last_clean_scrub_stamp":"2024-03-10T00:51:56.116216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T02:29:37.107341+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":29,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-10T00:52:38.019941+0000","last_change":"2024-03-10T00:51:53.119667+0000","last_active":"2024-03-10T00:52:38.019941+0000","last_peered":"2024-03-10T00:52:38.019941+0000","last_clean":"2024-03-10T00:52:38.019941+0000","last_became_active":"2024-03-10T00:51:53.119285+0000","last_became_peered":"2024-03-10T00:51:53.119285+0000","last_unstale":"2024-03-10T00:52:38.019941+0000","last_undegraded":"2024-03-10T00:52:38.019941+0000","last_fullsized":"2024-03-10T00:52:38.019941+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-10T00:51:52.085771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-10T00:51:52.085771+0000","last_clean_scrub_stamp":"2024-03-10T00:51:52.085771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T09:52:08.296964+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.d","version":"67'17","reported_seq":53,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-10T00:52:38.021283+0000","last_change":"2024-03-10T00:51:55.119216+0000","last_active":"2024-03-10T00:52:38.021283+0000","last_peered":"2024-03-10T00:52:38.021283+0000","last_clean":"2024-03-10T00:52:38.021283+0000","last_became_active":"2024-03-10T00:51:55.118921+0000","last_became_peered":"2024-03-10T00:51:55.118921+0000","last_unstale":"2024-03-10T00:52:38.021283+0000","last_undegraded":"2024-03-10T00:52:38.021283+0000","last_fullsized":"2024-03-10T00:52:38.021283+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-10T00:51:54.096929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-10T00:51:54.096929+0000","last_clean_scrub_stamp":"2024-03-10T00:51:54.096929+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T10:38:05.104385+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":19,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":17,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-10T00:52:38.021421+0000","last_change":"2024-03-10T00:51:59.139583+0000","last_active":"2024-03-10T00:52:38.021421+0000","last_peered":"2024-03-10T00:52:38.021421+0000","last_clean":"2024-03-10T00:52:38.021421+0000","last_became_active":"2024-03-10T00:51:59.139298+0000","last_became_peered":"2024-03-10T00:51:59.139298+0000","last_unstale":"2024-03-10T00:52:38.021421+0000","last_undegraded":"2024-03-10T00:52:38.021421+0000","last_fullsized":"2024-03-10T00:52:38.021421+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-10T00:51:58.121476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-10T00:51:58.121476+0000","last_clean_scrub_stamp":"2024-03-10T00:51:58.121476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T09:02:31.591573+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"67'11","reported_seq":50,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-10T00:53:13.173518+0000","last_change":"2024-03-10T00:51:57.136114+0000","last_active":"2024-03-10T00:53:13.173518+0000","last_peered":"2024-03-10T00:53:13.173518+0000","last_clean":"2024-03-10T00:53:13.173518+0000","last_became_active":"2024-03-10T00:51:57.135821+0000","last_became_peered":"2024-03-10T00:51:57.135821+0000","last_unstale":"2024-03-10T00:53:13.173518+0000","last_undegraded":"2024-03-10T00:53:13.173518+0000","last_fullsized":"2024-03-10T00:53:13.173518+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-10T00:51:56.116216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-10T00:51:56.116216+0000","last_clean_scrub_stamp":"2024-03-10T00:51:56.116216+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T01:52:41.238751+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":29,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-10T00:52:38.021547+0000","last_change":"2024-03-10T00:51:53.108796+0000","last_active":"2024-03-10T00:52:38.021547+0000","last_peered":"2024-03-10T00:52:38.021547+0000","last_clean":"2024-03-10T00:52:38.021547+0000","last_became_active":"2024-03-10T00:51:53.108395+0000","last_became_peered":"2024-03-10T00:51:53.108395+0000","last_unstale":"2024-03-10T00:52:38.021547+0000","last_undegraded":"2024-03-10T00:52:38.021547+0000","last_fullsized":"2024-03-10T00:52:38.021547+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-10T00:51:52.085771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-10T00:51:52.085771+0000","last_clean_scrub_stamp":"2024-03-10T00:51:52.085771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T12:14:51.940399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"67'10","reported_seq":40,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-10T00:52:38.020986+0000","last_change":"2024-03-10T00:51:55.117575+0000","last_active":"2024-03-10T00:52:38.020986+0000","last_peered":"2024-03-10T00:52:38.020986+0000","last_clean":"2024-03-10T00:52:38.020986+0000","last_became_active":"2024-03-10T00:51:55.117270+0000","last_became_peered":"2024-03-10T00:51:55.117270+0000","last_unstale":"2024-03-10T00:52:38.020986+0000","last_undegraded":"2024-03-10T00:52:38.020986+0000","last_fullsized":"2024-03-10T00:52:38.020986+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-10T00:51:54.096929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-10T00:51:54.096929+0000","last_clean_scrub_stamp":"2024-03-10T00:51:54.096929+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T08:32:41.786649+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":383,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1149,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":88,"ondisk_log_size":88,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":210,"num_object_clones":0,"num_object_copies":630,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":210,"num_whiteouts":0,"num_read":989,"num_read_kb":714,"num_write":609,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":591,"ondisk_log_size":591,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1457,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":34,"num_read_kb":34,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4371,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":96,"acting":96,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":242,"num_read_kb":333,"num_write":245,"num_write_kb":3982,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":188,"ondisk_log_size":188,"up":3,"acting":3,"num_store_stats":3},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":79,"num_read_kb":74,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":9,"acting":9,"num_store_stats":7}],"osd_stats":[{"osd":7,"up_from":54,"seq":231928234006,"num_pgs":53,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93741056,"kb_used":32520,"kb_used_data":1340,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93708536,"statfs":{"total":95990841344,"available":95957540864,"internally_reserved":0,"allocated":1372160,"data_stored":908965,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":46,"seq":197568495643,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93741056,"kb_used":28336,"kb_used_data":1320,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93712720,"statfs":{"total":95990841344,"available":95961825280,"internally_reserved":0,"allocated":1351680,"data_stored":907436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574902,"num_pgs":57,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93741056,"kb_used":31976,"kb_used_data":796,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93709080,"statfs":{"total":95990841344,"available":95958097920,"internally_reserved":0,"allocated":815104,"data_stored":318442,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738428,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93741056,"kb_used":32524,"kb_used_data":1344,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93708532,"statfs":{"total":95990841344,"available":95957536768,"internally_reserved":0,"allocated":1376256,"data_stored":909092,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411377,"num_pgs":36,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93741056,"kb_used":27748,"kb_used_data":728,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93713308,"statfs":{"total":95990841344,"available":95962427392,"internally_reserved":0,"allocated":745472,"data_stored":316772,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182444,"num_pgs":56,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93741056,"kb_used":31956,"kb_used_data":776,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93709100,"statfs":{"total":95990841344,"available":95958118400,"internally_reserved":0,"allocated":794624,"data_stored":318436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":30,"seq":128849018918,"num_pgs":58,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93741056,"kb_used":27780,"kb_used_data":760,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93713276,"statfs":{"total":95990841344,"available":95962394624,"internally_reserved":0,"allocated":778240,"data_stored":317262,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757282,"num_pgs":47,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93741056,"kb_used":31920,"kb_used_data":740,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93709136,"statfs":{"total":95990841344,"available":95958155264,"internally_reserved":0,"allocated":757760,"data_stored":317381,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1411,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":921,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":967,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":90112,"data_stored":2338,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":798,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":990,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-10T00:53:23.378 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:831bc188fa69f143bbba69ac76119e784500e8e3 shell --fsid 75c37498-de77-11ee-95c7-87774f69a715 -- ceph pg dump --format=json 2024-03-10T00:53:24.109 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/75c37498-de77-11ee-95c7-87774f69a715/mon.a/config 2024-03-10T00:53:25.088 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:53:24 smithi106 ceph-mon[25742]: pgmap v26: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-10T00:53:25.088 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:53:24 smithi106 ceph-mon[33460]: pgmap v26: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-10T00:53:25.230 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:53:24 smithi170 ceph-mon[27620]: pgmap v26: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-10T00:53:27.175 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:53:26 smithi106 ceph-mon[25742]: pgmap v27: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-10T00:53:27.175 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:53:26 smithi106 ceph-mon[33460]: pgmap v27: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-10T00:53:27.274 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:53:26 smithi170 ceph-mon[27620]: pgmap v27: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-10T00:53:27.496 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-03-10T00:53:27.498 INFO:teuthology.orchestra.run.smithi106.stderr:dumped all 2024-03-10T00:53:28.567 INFO:teuthology.orchestra.run.smithi106.stdout:{"pg_ready":true,"pg_map":{"version":27,"stamp":"2024-03-10T00:53:25.571769+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":596318,"num_objects":230,"num_object_clones":0,"num_object_copies":690,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":230,"num_whiteouts":0,"num_read":1352,"num_read_kb":1163,"num_write":873,"num_write_kb":4027,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":882,"ondisk_log_size":882,"up":396,"acting":396,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":396,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749928448,"kb_used":244760,"kb_used_data":7804,"kb_used_omap":12,"kb_used_meta":236851,"kb_avail":749683688,"statfs":{"total":767926730752,"available":767676096512,"internally_reserved":0,"allocated":7991296,"data_stored":4313786,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":242535984},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":15,"num_read_kb":15,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003021"},"pg_stats":[{"pgid":"6.1a","version":"0'0","reported_seq":17,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-10T00:52:38.020535+0000","last_change":"2024-03-10T00:51:59.144683+0000","last_active":"2024-03-10T00:52:38.020535+0000","last_peered":"2024-03-10T00:52:38.020535+0000","last_clean":"2024-03-10T00:52:38.020535+0000","last_became_active":"2024-03-10T00:51:59.144339+0000","last_became_peered":"2024-03-10T00:51:59.144339+0000","last_unstale":"2024-03-10T00:52:38.020535+0000","last_undegraded":"2024-03-10T00:52:38.020535+0000","last_fullsized":"2024-03-10T00:52:38.020535+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-10T00:51:58.121476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-10T00:51:58.121476+0000","last_clean_scrub_stamp":"2024-03-10T00:51:58.121476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T10:25:17.519277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":21,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-10T00:52:37.551023+0000","last_change":"2024-03-10T00:51:57.136590+0000","last_active":"2024-03-10T00:52:37.551023+0000","last_peered":"2024-03-10T00:52:37.551023+0000","last_clean":"2024-03-10T00:52:37.551023+0000","last_became_active":"2024-03-10T00:51:57.136304+0000","last_became_peered":"2024-03-10T00:51:57.136304+0000","last_unstale":"2024-03-10T00:52:37.551023+0000","last_undegraded":"2024-03-10T00:52:37.551023+0000","last_fullsized":"2024-03-10T00:52:37.551023+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-10T00:51:56.116216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-10T00:51:56.116216+0000","last_clean_scrub_stamp":"2024-03-10T00:51:56.116216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T11:23:56.466435+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":29,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-10T00:52:38.020997+0000","last_change":"2024-03-10T00:51:53.110073+0000","last_active":"2024-03-10T00:52:38.020997+0000","last_peered":"2024-03-10T00:52:38.020997+0000","last_clean":"2024-03-10T00:52:38.020997+0000","last_became_active":"2024-03-10T00:51:53.109784+0000","last_became_peered":"2024-03-10T00:51:53.109784+0000","last_unstale":"2024-03-10T00:52:38.020997+0000","last_undegraded":"2024-03-10T00:52:38.020997+0000","last_fullsized":"2024-03-10T00:52:38.020997+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-10T00:51:52.085771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-10T00:51:52.085771+0000","last_clean_scrub_stamp":"2024-03-10T00:51:52.085771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T12:15:50.790714+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"67'15","reported_seq":53,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-10T00:52:38.019958+0000","last_change":"2024-03-10T00:51:55.122281+0000","last_active":"2024-03-10T00:52:38.019958+0000","last_peered":"2024-03-10T00:52:38.019958+0000","last_clean":"2024-03-10T00:52:38.019958+0000","last_became_active":"2024-03-10T00:51:55.121983+0000","last_became_peered":"2024-03-10T00:51:55.121983+0000","last_unstale":"2024-03-10T00:52:38.019958+0000","last_undegraded":"2024-03-10T00:52:38.019958+0000","last_fullsized":"2024-03-10T00:52:38.019958+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-10T00:51:54.096929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-10T00:51:54.096929+0000","last_clean_scrub_stamp":"2024-03-10T00:51:54.096929+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T09:37:56.832827+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"67'1","reported_seq":18,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-10T00:52:38.021355+0000","last_change":"2024-03-10T00:51:59.266237+0000","last_active":"2024-03-10T00:52:38.021355+0000","last_peered":"2024-03-10T00:52:38.021355+0000","last_clean":"2024-03-10T00:52:38.021355+0000","last_became_active":"2024-03-10T00:51:59.265827+0000","last_became_peered":"2024-03-10T00:51:59.265827+0000","last_unstale":"2024-03-10T00:52:38.021355+0000","last_undegraded":"2024-03-10T00:52:38.021355+0000","last_fullsized":"2024-03-10T00:52:38.021355+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-10T00:51:58.121476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-10T00:51:58.121476+0000","last_clean_scrub_stamp":"2024-03-10T00:51:58.121476+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T08:53:28.833221+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":21,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-10T00:52:38.020092+0000","last_change":"2024-03-10T00:51:57.132759+0000","last_active":"2024-03-10T00:52:38.020092+0000","last_peered":"2024-03-10T00:52:38.020092+0000","last_clean":"2024-03-10T00:52:38.020092+0000","last_became_active":"2024-03-10T00:51:57.132450+0000","last_became_peered":"2024-03-10T00:51:57.132450+0000","last_unstale":"2024-03-10T00:52:38.020092+0000","last_undegraded":"2024-03-10T00:52:38.020092+0000","last_fullsized":"2024-03-10T00:52:38.020092+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-10T00:51:56.116216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-10T00:51:56.116216+0000","last_clean_scrub_stamp":"2024-03-10T00:51:56.116216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T10:05:25.513309+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":29,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-10T00:52:38.022580+0000","last_change":"2024-03-10T00:51:53.111428+0000","last_active":"2024-03-10T00:52:38.022580+0000","last_peered":"2024-03-10T00:52:38.022580+0000","last_clean":"2024-03-10T00:52:38.022580+0000","last_became_active":"2024-03-10T00:51:53.108723+0000","last_became_peered":"2024-03-10T00:51:53.108723+0000","last_unstale":"2024-03-10T00:52:38.022580+0000","last_undegraded":"2024-03-10T00:52:38.022580+0000","last_fullsized":"2024-03-10T00:52:38.022580+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-10T00:51:52.085771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-10T00:51:52.085771+0000","last_clean_scrub_stamp":"2024-03-10T00:51:52.085771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T01:12:53.162235+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"67'41","reported_seq":100,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-10T00:52:38.020948+0000","last_change":"2024-03-10T00:51:55.115459+0000","last_active":"2024-03-10T00:52:38.020948+0000","last_peered":"2024-03-10T00:52:38.020948+0000","last_clean":"2024-03-10T00:52:38.020948+0000","last_became_active":"2024-03-10T00:51:55.115184+0000","last_became_peered":"2024-03-10T00:51:55.115184+0000","last_unstale":"2024-03-10T00:52:38.020948+0000","last_undegraded":"2024-03-10T00:52:38.020948+0000","last_fullsized":"2024-03-10T00:52:38.020948+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-10T00:51:54.096929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-10T00:51:54.096929+0000","last_clean_scrub_stamp":"2024-03-10T00:51:54.096929+0000","objects_scrubbed":0,"log_size":41,"log_dups_size":0,"ondisk_log_size":41,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T05:39:59.836491+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":62,"num_read_kb":49,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":17,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-10T00:52:38.020011+0000","last_change":"2024-03-10T00:51:59.144757+0000","last_active":"2024-03-10T00:52:38.020011+0000","last_peered":"2024-03-10T00:52:38.020011+0000","last_clean":"2024-03-10T00:52:38.020011+0000","last_became_active":"2024-03-10T00:51:59.144453+0000","last_became_peered":"2024-03-10T00:51:59.144453+0000","last_unstale":"2024-03-10T00:52:38.020011+0000","last_undegraded":"2024-03-10T00:52:38.020011+0000","last_fullsized":"2024-03-10T00:52:38.020011+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-10T00:51:58.121476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-10T00:51:58.121476+0000","last_clean_scrub_stamp":"2024-03-10T00:51:58.121476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T05:08:50.523240+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":21,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-10T00:52:37.552210+0000","last_change":"2024-03-10T00:51:57.137078+0000","last_active":"2024-03-10T00:52:37.552210+0000","last_peered":"2024-03-10T00:52:37.552210+0000","last_clean":"2024-03-10T00:52:37.552210+0000","last_became_active":"2024-03-10T00:51:57.136723+0000","last_became_peered":"2024-03-10T00:51:57.136723+0000","last_unstale":"2024-03-10T00:52:37.552210+0000","last_undegraded":"2024-03-10T00:52:37.552210+0000","last_fullsized":"2024-03-10T00:52:37.552210+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-10T00:51:56.116216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-10T00:51:56.116216+0000","last_clean_scrub_stamp":"2024-03-10T00:51:56.116216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T07:12:56.594081+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":29,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-10T00:52:37.553141+0000","last_change":"2024-03-10T00:51:53.114545+0000","last_active":"2024-03-10T00:52:37.553141+0000","last_peered":"2024-03-10T00:52:37.553141+0000","last_clean":"2024-03-10T00:52:37.553141+0000","last_became_active":"2024-03-10T00:51:53.114231+0000","last_became_peered":"2024-03-10T00:51:53.114231+0000","last_unstale":"2024-03-10T00:52:37.553141+0000","last_undegraded":"2024-03-10T00:52:37.553141+0000","last_fullsized":"2024-03-10T00:52:37.553141+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-10T00:51:52.085771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-10T00:51:52.085771+0000","last_clean_scrub_stamp":"2024-03-10T00:51:52.085771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T10:13:15.469773+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"67'21","reported_seq":65,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-10T00:52:38.020683+0000","last_change":"2024-03-10T00:51:55.113106+0000","last_active":"2024-03-10T00:52:38.020683+0000","last_peered":"2024-03-10T00:52:38.020683+0000","last_clean":"2024-03-10T00:52:38.020683+0000","last_became_active":"2024-03-10T00:51:55.112804+0000","last_became_peered":"2024-03-10T00:51:55.112804+0000","last_unstale":"2024-03-10T00:52:38.020683+0000","last_undegraded":"2024-03-10T00:52:38.020683+0000","last_fullsized":"2024-03-10T00:52:38.020683+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-10T00:51:54.096929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-10T00:51:54.096929+0000","last_clean_scrub_stamp":"2024-03-10T00:51:54.096929+0000","objects_scrubbed":0,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T01:08:00.360628+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":35,"num_read_kb":27,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":17,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-10T00:52:37.551907+0000","last_change":"2024-03-10T00:51:59.141885+0000","last_active":"2024-03-10T00:52:37.551907+0000","last_peered":"2024-03-10T00:52:37.551907+0000","last_clean":"2024-03-10T00:52:37.551907+0000","last_became_active":"2024-03-10T00:51:59.141598+0000","last_became_peered":"2024-03-10T00:51:59.141598+0000","last_unstale":"2024-03-10T00:52:37.551907+0000","last_undegraded":"2024-03-10T00:52:37.551907+0000","last_fullsized":"2024-03-10T00:52:37.551907+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-10T00:51:58.121476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-10T00:51:58.121476+0000","last_clean_scrub_stamp":"2024-03-10T00:51:58.121476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T08:31:42.089353+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":21,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-10T00:52:37.662324+0000","last_change":"2024-03-10T00:51:57.139310+0000","last_active":"2024-03-10T00:52:37.662324+0000","last_peered":"2024-03-10T00:52:37.662324+0000","last_clean":"2024-03-10T00:52:37.662324+0000","last_became_active":"2024-03-10T00:51:57.139029+0000","last_became_peered":"2024-03-10T00:51:57.139029+0000","last_unstale":"2024-03-10T00:52:37.662324+0000","last_undegraded":"2024-03-10T00:52:37.662324+0000","last_fullsized":"2024-03-10T00:52:37.662324+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-10T00:51:56.116216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-10T00:51:56.116216+0000","last_clean_scrub_stamp":"2024-03-10T00:51:56.116216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T10:37:53.823563+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":29,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-10T00:52:37.552390+0000","last_change":"2024-03-10T00:51:53.113829+0000","last_active":"2024-03-10T00:52:37.552390+0000","last_peered":"2024-03-10T00:52:37.552390+0000","last_clean":"2024-03-10T00:52:37.552390+0000","last_became_active":"2024-03-10T00:51:53.113552+0000","last_became_peered":"2024-03-10T00:51:53.113552+0000","last_unstale":"2024-03-10T00:52:37.552390+0000","last_undegraded":"2024-03-10T00:52:37.552390+0000","last_fullsized":"2024-03-10T00:52:37.552390+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-10T00:51:52.085771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-10T00:51:52.085771+0000","last_clean_scrub_stamp":"2024-03-10T00:51:52.085771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T01:51:27.567647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"67'11","reported_seq":47,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-10T00:52:38.020908+0000","last_change":"2024-03-10T00:51:55.120009+0000","last_active":"2024-03-10T00:52:38.020908+0000","last_peered":"2024-03-10T00:52:38.020908+0000","last_clean":"2024-03-10T00:52:38.020908+0000","last_became_active":"2024-03-10T00:51:55.119702+0000","last_became_peered":"2024-03-10T00:51:55.119702+0000","last_unstale":"2024-03-10T00:52:38.020908+0000","last_undegraded":"2024-03-10T00:52:38.020908+0000","last_fullsized":"2024-03-10T00:52:38.020908+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-10T00:51:54.096929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-10T00:51:54.096929+0000","last_clean_scrub_stamp":"2024-03-10T00:51:54.096929+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T06:14:04.512628+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":20,"num_read_kb":15,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":17,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-10T00:52:38.021320+0000","last_change":"2024-03-10T00:51:59.263679+0000","last_active":"2024-03-10T00:52:38.021320+0000","last_peered":"2024-03-10T00:52:38.021320+0000","last_clean":"2024-03-10T00:52:38.021320+0000","last_became_active":"2024-03-10T00:51:59.263348+0000","last_became_peered":"2024-03-10T00:51:59.263348+0000","last_unstale":"2024-03-10T00:52:38.021320+0000","last_undegraded":"2024-03-10T00:52:38.021320+0000","last_fullsized":"2024-03-10T00:52:38.021320+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-10T00:51:58.121476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-10T00:51:58.121476+0000","last_clean_scrub_stamp":"2024-03-10T00:51:58.121476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T07:09:44.346803+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":21,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-10T00:52:37.552017+0000","last_change":"2024-03-10T00:51:57.135032+0000","last_active":"2024-03-10T00:52:37.552017+0000","last_peered":"2024-03-10T00:52:37.552017+0000","last_clean":"2024-03-10T00:52:37.552017+0000","last_became_active":"2024-03-10T00:51:57.134727+0000","last_became_peered":"2024-03-10T00:51:57.134727+0000","last_unstale":"2024-03-10T00:52:37.552017+0000","last_undegraded":"2024-03-10T00:52:37.552017+0000","last_fullsized":"2024-03-10T00:52:37.552017+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-10T00:51:56.116216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-10T00:51:56.116216+0000","last_clean_scrub_stamp":"2024-03-10T00:51:56.116216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T11:32:03.708915+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":29,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-10T00:52:38.022041+0000","last_change":"2024-03-10T00:51:53.114924+0000","last_active":"2024-03-10T00:52:38.022041+0000","last_peered":"2024-03-10T00:52:38.022041+0000","last_clean":"2024-03-10T00:52:38.022041+0000","last_became_active":"2024-03-10T00:51:53.114633+0000","last_became_peered":"2024-03-10T00:51:53.114633+0000","last_unstale":"2024-03-10T00:52:38.022041+0000","last_undegraded":"2024-03-10T00:52:38.022041+0000","last_fullsized":"2024-03-10T00:52:38.022041+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-10T00:51:52.085771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-10T00:51:52.085771+0000","last_clean_scrub_stamp":"2024-03-10T00:51:52.085771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T11:34:53.070630+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"67'15","reported_seq":50,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-10T00:52:37.555249+0000","last_change":"2024-03-10T00:51:55.118445+0000","last_active":"2024-03-10T00:52:37.555249+0000","last_peered":"2024-03-10T00:52:37.555249+0000","last_clean":"2024-03-10T00:52:37.555249+0000","last_became_active":"2024-03-10T00:51:55.118151+0000","last_became_peered":"2024-03-10T00:51:55.118151+0000","last_unstale":"2024-03-10T00:52:37.555249+0000","last_undegraded":"2024-03-10T00:52:37.555249+0000","last_fullsized":"2024-03-10T00:52:37.555249+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-10T00:51:54.096929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-10T00:51:54.096929+0000","last_clean_scrub_stamp":"2024-03-10T00:51:54.096929+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T05:13:26.527752+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":17,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-10T00:52:38.022700+0000","last_change":"2024-03-10T00:51:59.263815+0000","last_active":"2024-03-10T00:52:38.022700+0000","last_peered":"2024-03-10T00:52:38.022700+0000","last_clean":"2024-03-10T00:52:38.022700+0000","last_became_active":"2024-03-10T00:51:59.263469+0000","last_became_peered":"2024-03-10T00:51:59.263469+0000","last_unstale":"2024-03-10T00:52:38.022700+0000","last_undegraded":"2024-03-10T00:52:38.022700+0000","last_fullsized":"2024-03-10T00:52:38.022700+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-10T00:51:58.121476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-10T00:51:58.121476+0000","last_clean_scrub_stamp":"2024-03-10T00:51:58.121476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T08:33:25.589194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":21,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-10T00:52:38.020920+0000","last_change":"2024-03-10T00:51:57.135717+0000","last_active":"2024-03-10T00:52:38.020920+0000","last_peered":"2024-03-10T00:52:38.020920+0000","last_clean":"2024-03-10T00:52:38.020920+0000","last_became_active":"2024-03-10T00:51:57.135284+0000","last_became_peered":"2024-03-10T00:51:57.135284+0000","last_unstale":"2024-03-10T00:52:38.020920+0000","last_undegraded":"2024-03-10T00:52:38.020920+0000","last_fullsized":"2024-03-10T00:52:38.020920+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-10T00:51:56.116216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-10T00:51:56.116216+0000","last_clean_scrub_stamp":"2024-03-10T00:51:56.116216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T01:53:23.572680+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":29,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-10T00:52:38.020708+0000","last_change":"2024-03-10T00:51:53.106328+0000","last_active":"2024-03-10T00:52:38.020708+0000","last_peered":"2024-03-10T00:52:38.020708+0000","last_clean":"2024-03-10T00:52:38.020708+0000","last_became_active":"2024-03-10T00:51:53.105960+0000","last_became_peered":"2024-03-10T00:51:53.105960+0000","last_unstale":"2024-03-10T00:52:38.020708+0000","last_undegraded":"2024-03-10T00:52:38.020708+0000","last_fullsized":"2024-03-10T00:52:38.020708+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-10T00:51:52.085771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-10T00:51:52.085771+0000","last_clean_scrub_stamp":"2024-03-10T00:51:52.085771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T08:41:09.650739+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"67'18","reported_seq":60,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-10T00:52:38.022465+0000","last_change":"2024-03-10T00:51:55.121368+0000","last_active":"2024-03-10T00:52:38.022465+0000","last_peered":"2024-03-10T00:52:38.022465+0000","last_clean":"2024-03-10T00:52:38.022465+0000","last_became_active":"2024-03-10T00:51:55.120985+0000","last_became_peered":"2024-03-10T00:51:55.120985+0000","last_unstale":"2024-03-10T00:52:38.022465+0000","last_undegraded":"2024-03-10T00:52:38.022465+0000","last_fullsized":"2024-03-10T00:52:38.022465+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-10T00:51:54.096929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-10T00:51:54.096929+0000","last_clean_scrub_stamp":"2024-03-10T00:51:54.096929+0000","objects_scrubbed":0,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T11:17:42.946770+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":34,"num_read_kb":24,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1c","version":"67'1","reported_seq":19,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-10T00:52:37.662548+0000","last_change":"2024-03-10T00:51:59.135745+0000","last_active":"2024-03-10T00:52:37.662548+0000","last_peered":"2024-03-10T00:52:37.662548+0000","last_clean":"2024-03-10T00:52:37.662548+0000","last_became_active":"2024-03-10T00:51:59.135464+0000","last_became_peered":"2024-03-10T00:51:59.135464+0000","last_unstale":"2024-03-10T00:52:37.662548+0000","last_undegraded":"2024-03-10T00:52:37.662548+0000","last_fullsized":"2024-03-10T00:52:37.662548+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-10T00:51:58.121476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-10T00:51:58.121476+0000","last_clean_scrub_stamp":"2024-03-10T00:51:58.121476+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T03:10:24.567775+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":370,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1f","version":"67'11","reported_seq":54,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-10T00:53:18.174712+0000","last_change":"2024-03-10T00:51:57.137284+0000","last_active":"2024-03-10T00:53:18.174712+0000","last_peered":"2024-03-10T00:53:18.174712+0000","last_clean":"2024-03-10T00:53:18.174712+0000","last_became_active":"2024-03-10T00:51:57.136953+0000","last_became_peered":"2024-03-10T00:51:57.136953+0000","last_unstale":"2024-03-10T00:53:18.174712+0000","last_undegraded":"2024-03-10T00:53:18.174712+0000","last_fullsized":"2024-03-10T00:53:18.174712+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-10T00:51:56.116216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-10T00:51:56.116216+0000","last_clean_scrub_stamp":"2024-03-10T00:51:56.116216+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T07:56:27.964307+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.19","version":"60'1","reported_seq":30,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-10T00:52:37.551149+0000","last_change":"2024-03-10T00:51:53.110576+0000","last_active":"2024-03-10T00:52:37.551149+0000","last_peered":"2024-03-10T00:52:37.551149+0000","last_clean":"2024-03-10T00:52:37.551149+0000","last_became_active":"2024-03-10T00:51:53.110183+0000","last_became_peered":"2024-03-10T00:51:53.110183+0000","last_unstale":"2024-03-10T00:52:37.551149+0000","last_undegraded":"2024-03-10T00:52:37.551149+0000","last_fullsized":"2024-03-10T00:52:37.551149+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-10T00:51:52.085771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-10T00:51:52.085771+0000","last_clean_scrub_stamp":"2024-03-10T00:51:52.085771+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T09:07:47.930342+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"67'16","reported_seq":52,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-10T00:52:38.021602+0000","last_change":"2024-03-10T00:51:55.114063+0000","last_active":"2024-03-10T00:52:38.021602+0000","last_peered":"2024-03-10T00:52:38.021602+0000","last_clean":"2024-03-10T00:52:38.021602+0000","last_became_active":"2024-03-10T00:51:55.113695+0000","last_became_peered":"2024-03-10T00:51:55.113695+0000","last_unstale":"2024-03-10T00:52:38.021602+0000","last_undegraded":"2024-03-10T00:52:38.021602+0000","last_fullsized":"2024-03-10T00:52:38.021602+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-10T00:51:54.096929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-10T00:51:54.096929+0000","last_clean_scrub_stamp":"2024-03-10T00:51:54.096929+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T04:41:10.790916+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":18,"num_write":16,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":17,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-10T00:52:37.552578+0000","last_change":"2024-03-10T00:51:59.140420+0000","last_active":"2024-03-10T00:52:37.552578+0000","last_peered":"2024-03-10T00:52:37.552578+0000","last_clean":"2024-03-10T00:52:37.552578+0000","last_became_active":"2024-03-10T00:51:59.140081+0000","last_became_peered":"2024-03-10T00:51:59.140081+0000","last_unstale":"2024-03-10T00:52:37.552578+0000","last_undegraded":"2024-03-10T00:52:37.552578+0000","last_fullsized":"2024-03-10T00:52:37.552578+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-10T00:51:58.121476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-10T00:51:58.121476+0000","last_clean_scrub_stamp":"2024-03-10T00:51:58.121476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T09:57:19.697840+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":21,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-10T00:52:38.021902+0000","last_change":"2024-03-10T00:51:57.134314+0000","last_active":"2024-03-10T00:52:38.021902+0000","last_peered":"2024-03-10T00:52:38.021902+0000","last_clean":"2024-03-10T00:52:38.021902+0000","last_became_active":"2024-03-10T00:51:57.133999+0000","last_became_peered":"2024-03-10T00:51:57.133999+0000","last_unstale":"2024-03-10T00:52:38.021902+0000","last_undegraded":"2024-03-10T00:52:38.021902+0000","last_fullsized":"2024-03-10T00:52:38.021902+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-10T00:51:56.116216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-10T00:51:56.116216+0000","last_clean_scrub_stamp":"2024-03-10T00:51:56.116216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T11:32:16.620975+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":29,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-10T00:52:38.020186+0000","last_change":"2024-03-10T00:51:53.114064+0000","last_active":"2024-03-10T00:52:38.020186+0000","last_peered":"2024-03-10T00:52:38.020186+0000","last_clean":"2024-03-10T00:52:38.020186+0000","last_became_active":"2024-03-10T00:51:53.113309+0000","last_became_peered":"2024-03-10T00:51:53.113309+0000","last_unstale":"2024-03-10T00:52:38.020186+0000","last_undegraded":"2024-03-10T00:52:38.020186+0000","last_fullsized":"2024-03-10T00:52:38.020186+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-10T00:51:52.085771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-10T00:51:52.085771+0000","last_clean_scrub_stamp":"2024-03-10T00:51:52.085771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T01:44:52.360919+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"67'17","reported_seq":56,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-10T00:52:38.020609+0000","last_change":"2024-03-10T00:51:55.119788+0000","last_active":"2024-03-10T00:52:38.020609+0000","last_peered":"2024-03-10T00:52:38.020609+0000","last_clean":"2024-03-10T00:52:38.020609+0000","last_became_active":"2024-03-10T00:51:55.119563+0000","last_became_peered":"2024-03-10T00:51:55.119563+0000","last_unstale":"2024-03-10T00:52:38.020609+0000","last_undegraded":"2024-03-10T00:52:38.020609+0000","last_fullsized":"2024-03-10T00:52:38.020609+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-10T00:51:54.096929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-10T00:51:54.096929+0000","last_clean_scrub_stamp":"2024-03-10T00:51:54.096929+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T08:26:59.664526+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":29,"num_read_kb":21,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":17,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-10T00:52:37.662319+0000","last_change":"2024-03-10T00:51:59.134997+0000","last_active":"2024-03-10T00:52:37.662319+0000","last_peered":"2024-03-10T00:52:37.662319+0000","last_clean":"2024-03-10T00:52:37.662319+0000","last_became_active":"2024-03-10T00:51:59.134743+0000","last_became_peered":"2024-03-10T00:51:59.134743+0000","last_unstale":"2024-03-10T00:52:37.662319+0000","last_undegraded":"2024-03-10T00:52:37.662319+0000","last_fullsized":"2024-03-10T00:52:37.662319+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-10T00:51:58.121476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-10T00:51:58.121476+0000","last_clean_scrub_stamp":"2024-03-10T00:51:58.121476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T06:56:54.895354+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":21,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-10T00:52:38.019907+0000","last_change":"2024-03-10T00:51:57.139157+0000","last_active":"2024-03-10T00:52:38.019907+0000","last_peered":"2024-03-10T00:52:38.019907+0000","last_clean":"2024-03-10T00:52:38.019907+0000","last_became_active":"2024-03-10T00:51:57.138839+0000","last_became_peered":"2024-03-10T00:51:57.138839+0000","last_unstale":"2024-03-10T00:52:38.019907+0000","last_undegraded":"2024-03-10T00:52:38.019907+0000","last_fullsized":"2024-03-10T00:52:38.019907+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-10T00:51:56.116216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-10T00:51:56.116216+0000","last_clean_scrub_stamp":"2024-03-10T00:51:56.116216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T10:35:01.904822+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":29,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-10T00:52:38.020918+0000","last_change":"2024-03-10T00:51:53.109936+0000","last_active":"2024-03-10T00:52:38.020918+0000","last_peered":"2024-03-10T00:52:38.020918+0000","last_clean":"2024-03-10T00:52:38.020918+0000","last_became_active":"2024-03-10T00:51:53.109660+0000","last_became_peered":"2024-03-10T00:51:53.109660+0000","last_unstale":"2024-03-10T00:52:38.020918+0000","last_undegraded":"2024-03-10T00:52:38.020918+0000","last_fullsized":"2024-03-10T00:52:38.020918+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-10T00:51:52.085771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-10T00:51:52.085771+0000","last_clean_scrub_stamp":"2024-03-10T00:51:52.085771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T02:55:56.270302+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"67'4","reported_seq":31,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-10T00:52:38.020287+0000","last_change":"2024-03-10T00:51:55.120579+0000","last_active":"2024-03-10T00:52:38.020287+0000","last_peered":"2024-03-10T00:52:38.020287+0000","last_clean":"2024-03-10T00:52:38.020287+0000","last_became_active":"2024-03-10T00:51:55.120301+0000","last_became_peered":"2024-03-10T00:51:55.120301+0000","last_unstale":"2024-03-10T00:52:38.020287+0000","last_undegraded":"2024-03-10T00:52:38.020287+0000","last_fullsized":"2024-03-10T00:52:38.020287+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-10T00:51:54.096929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-10T00:51:54.096929+0000","last_clean_scrub_stamp":"2024-03-10T00:51:54.096929+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T01:56:09.559040+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":17,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-10T00:52:38.021162+0000","last_change":"2024-03-10T00:51:59.263814+0000","last_active":"2024-03-10T00:52:38.021162+0000","last_peered":"2024-03-10T00:52:38.021162+0000","last_clean":"2024-03-10T00:52:38.021162+0000","last_became_active":"2024-03-10T00:51:59.263443+0000","last_became_peered":"2024-03-10T00:51:59.263443+0000","last_unstale":"2024-03-10T00:52:38.021162+0000","last_undegraded":"2024-03-10T00:52:38.021162+0000","last_fullsized":"2024-03-10T00:52:38.021162+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-10T00:51:58.121476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-10T00:51:58.121476+0000","last_clean_scrub_stamp":"2024-03-10T00:51:58.121476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T07:11:49.745278+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":21,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-10T00:52:37.662516+0000","last_change":"2024-03-10T00:51:57.138684+0000","last_active":"2024-03-10T00:52:37.662516+0000","last_peered":"2024-03-10T00:52:37.662516+0000","last_clean":"2024-03-10T00:52:37.662516+0000","last_became_active":"2024-03-10T00:51:57.138398+0000","last_became_peered":"2024-03-10T00:51:57.138398+0000","last_unstale":"2024-03-10T00:52:37.662516+0000","last_undegraded":"2024-03-10T00:52:37.662516+0000","last_fullsized":"2024-03-10T00:52:37.662516+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-10T00:51:56.116216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-10T00:51:56.116216+0000","last_clean_scrub_stamp":"2024-03-10T00:51:56.116216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T01:54:53.914173+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":29,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-10T00:52:37.552499+0000","last_change":"2024-03-10T00:51:53.113570+0000","last_active":"2024-03-10T00:52:37.552499+0000","last_peered":"2024-03-10T00:52:37.552499+0000","last_clean":"2024-03-10T00:52:37.552499+0000","last_became_active":"2024-03-10T00:51:53.113228+0000","last_became_peered":"2024-03-10T00:51:53.113228+0000","last_unstale":"2024-03-10T00:52:37.552499+0000","last_undegraded":"2024-03-10T00:52:37.552499+0000","last_fullsized":"2024-03-10T00:52:37.552499+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-10T00:51:52.085771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-10T00:51:52.085771+0000","last_clean_scrub_stamp":"2024-03-10T00:51:52.085771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T08:39:17.810891+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.11","version":"67'17","reported_seq":56,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-10T00:52:38.020801+0000","last_change":"2024-03-10T00:51:55.117051+0000","last_active":"2024-03-10T00:52:38.020801+0000","last_peered":"2024-03-10T00:52:38.020801+0000","last_clean":"2024-03-10T00:52:38.020801+0000","last_became_active":"2024-03-10T00:51:55.116157+0000","last_became_peered":"2024-03-10T00:51:55.116157+0000","last_unstale":"2024-03-10T00:52:38.020801+0000","last_undegraded":"2024-03-10T00:52:38.020801+0000","last_fullsized":"2024-03-10T00:52:38.020801+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-10T00:51:54.096929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-10T00:51:54.096929+0000","last_clean_scrub_stamp":"2024-03-10T00:51:54.096929+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T06:27:53.033220+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":29,"num_read_kb":21,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":17,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-10T00:52:38.019769+0000","last_change":"2024-03-10T00:51:59.147493+0000","last_active":"2024-03-10T00:52:38.019769+0000","last_peered":"2024-03-10T00:52:38.019769+0000","last_clean":"2024-03-10T00:52:38.019769+0000","last_became_active":"2024-03-10T00:51:59.146635+0000","last_became_peered":"2024-03-10T00:51:59.146635+0000","last_unstale":"2024-03-10T00:52:38.019769+0000","last_undegraded":"2024-03-10T00:52:38.019769+0000","last_fullsized":"2024-03-10T00:52:38.019769+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-10T00:51:58.121476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-10T00:51:58.121476+0000","last_clean_scrub_stamp":"2024-03-10T00:51:58.121476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T10:13:21.568724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":21,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-10T00:52:38.021462+0000","last_change":"2024-03-10T00:51:57.137386+0000","last_active":"2024-03-10T00:52:38.021462+0000","last_peered":"2024-03-10T00:52:38.021462+0000","last_clean":"2024-03-10T00:52:38.021462+0000","last_became_active":"2024-03-10T00:51:57.137037+0000","last_became_peered":"2024-03-10T00:51:57.137037+0000","last_unstale":"2024-03-10T00:52:38.021462+0000","last_undegraded":"2024-03-10T00:52:38.021462+0000","last_fullsized":"2024-03-10T00:52:38.021462+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-10T00:51:56.116216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-10T00:51:56.116216+0000","last_clean_scrub_stamp":"2024-03-10T00:51:56.116216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T07:10:20.789695+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":29,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-10T00:52:37.662529+0000","last_change":"2024-03-10T00:51:53.118133+0000","last_active":"2024-03-10T00:52:37.662529+0000","last_peered":"2024-03-10T00:52:37.662529+0000","last_clean":"2024-03-10T00:52:37.662529+0000","last_became_active":"2024-03-10T00:51:53.117852+0000","last_became_peered":"2024-03-10T00:51:53.117852+0000","last_unstale":"2024-03-10T00:52:37.662529+0000","last_undegraded":"2024-03-10T00:52:37.662529+0000","last_fullsized":"2024-03-10T00:52:37.662529+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-10T00:51:52.085771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-10T00:51:52.085771+0000","last_clean_scrub_stamp":"2024-03-10T00:51:52.085771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T12:29:02.349185+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.12","version":"67'9","reported_seq":41,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-10T00:52:37.551868+0000","last_change":"2024-03-10T00:51:55.119656+0000","last_active":"2024-03-10T00:52:37.551868+0000","last_peered":"2024-03-10T00:52:37.551868+0000","last_clean":"2024-03-10T00:52:37.551868+0000","last_became_active":"2024-03-10T00:51:55.119386+0000","last_became_peered":"2024-03-10T00:51:55.119386+0000","last_unstale":"2024-03-10T00:52:37.551868+0000","last_undegraded":"2024-03-10T00:52:37.551868+0000","last_fullsized":"2024-03-10T00:52:37.551868+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-10T00:51:54.096929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-10T00:51:54.096929+0000","last_clean_scrub_stamp":"2024-03-10T00:51:54.096929+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T03:36:40.047496+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":17,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-10T00:52:38.020131+0000","last_change":"2024-03-10T00:51:59.143347+0000","last_active":"2024-03-10T00:52:38.020131+0000","last_peered":"2024-03-10T00:52:38.020131+0000","last_clean":"2024-03-10T00:52:38.020131+0000","last_became_active":"2024-03-10T00:51:59.143014+0000","last_became_peered":"2024-03-10T00:51:59.143014+0000","last_unstale":"2024-03-10T00:52:38.020131+0000","last_undegraded":"2024-03-10T00:52:38.020131+0000","last_fullsized":"2024-03-10T00:52:38.020131+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-10T00:51:58.121476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-10T00:51:58.121476+0000","last_clean_scrub_stamp":"2024-03-10T00:51:58.121476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T08:22:30.688743+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":21,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-10T00:52:37.551632+0000","last_change":"2024-03-10T00:51:57.138450+0000","last_active":"2024-03-10T00:52:37.551632+0000","last_peered":"2024-03-10T00:52:37.551632+0000","last_clean":"2024-03-10T00:52:37.551632+0000","last_became_active":"2024-03-10T00:51:57.138081+0000","last_became_peered":"2024-03-10T00:51:57.138081+0000","last_unstale":"2024-03-10T00:52:37.551632+0000","last_undegraded":"2024-03-10T00:52:37.551632+0000","last_fullsized":"2024-03-10T00:52:37.551632+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-10T00:51:56.116216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-10T00:51:56.116216+0000","last_clean_scrub_stamp":"2024-03-10T00:51:56.116216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T08:53:57.773539+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":29,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-10T00:52:38.021257+0000","last_change":"2024-03-10T00:51:53.113221+0000","last_active":"2024-03-10T00:52:38.021257+0000","last_peered":"2024-03-10T00:52:38.021257+0000","last_clean":"2024-03-10T00:52:38.021257+0000","last_became_active":"2024-03-10T00:51:53.112943+0000","last_became_peered":"2024-03-10T00:51:53.112943+0000","last_unstale":"2024-03-10T00:52:38.021257+0000","last_undegraded":"2024-03-10T00:52:38.021257+0000","last_fullsized":"2024-03-10T00:52:38.021257+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-10T00:51:52.085771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-10T00:51:52.085771+0000","last_clean_scrub_stamp":"2024-03-10T00:51:52.085771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T03:13:22.694962+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"67'11","reported_seq":44,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-10T00:52:38.021027+0000","last_change":"2024-03-10T00:51:55.120437+0000","last_active":"2024-03-10T00:52:38.021027+0000","last_peered":"2024-03-10T00:52:38.021027+0000","last_clean":"2024-03-10T00:52:38.021027+0000","last_became_active":"2024-03-10T00:51:55.120154+0000","last_became_peered":"2024-03-10T00:51:55.120154+0000","last_unstale":"2024-03-10T00:52:38.021027+0000","last_undegraded":"2024-03-10T00:52:38.021027+0000","last_fullsized":"2024-03-10T00:52:38.021027+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-10T00:51:54.096929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-10T00:51:54.096929+0000","last_clean_scrub_stamp":"2024-03-10T00:51:54.096929+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T02:29:54.442646+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":17,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-10T00:52:38.021292+0000","last_change":"2024-03-10T00:51:59.147162+0000","last_active":"2024-03-10T00:52:38.021292+0000","last_peered":"2024-03-10T00:52:38.021292+0000","last_clean":"2024-03-10T00:52:38.021292+0000","last_became_active":"2024-03-10T00:51:59.145954+0000","last_became_peered":"2024-03-10T00:51:59.145954+0000","last_unstale":"2024-03-10T00:52:38.021292+0000","last_undegraded":"2024-03-10T00:52:38.021292+0000","last_fullsized":"2024-03-10T00:52:38.021292+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-10T00:51:58.121476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-10T00:51:58.121476+0000","last_clean_scrub_stamp":"2024-03-10T00:51:58.121476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T06:59:05.277266+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.15","version":"67'11","reported_seq":51,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-10T00:53:18.174728+0000","last_change":"2024-03-10T00:51:57.138389+0000","last_active":"2024-03-10T00:53:18.174728+0000","last_peered":"2024-03-10T00:53:18.174728+0000","last_clean":"2024-03-10T00:53:18.174728+0000","last_became_active":"2024-03-10T00:51:57.138045+0000","last_became_peered":"2024-03-10T00:51:57.138045+0000","last_unstale":"2024-03-10T00:53:18.174728+0000","last_undegraded":"2024-03-10T00:53:18.174728+0000","last_fullsized":"2024-03-10T00:53:18.174728+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-10T00:51:56.116216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-10T00:51:56.116216+0000","last_clean_scrub_stamp":"2024-03-10T00:51:56.116216+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T12:27:13.728540+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":29,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-10T00:52:37.662545+0000","last_change":"2024-03-10T00:51:53.115683+0000","last_active":"2024-03-10T00:52:37.662545+0000","last_peered":"2024-03-10T00:52:37.662545+0000","last_clean":"2024-03-10T00:52:37.662545+0000","last_became_active":"2024-03-10T00:51:53.115238+0000","last_became_peered":"2024-03-10T00:51:53.115238+0000","last_unstale":"2024-03-10T00:52:37.662545+0000","last_undegraded":"2024-03-10T00:52:37.662545+0000","last_fullsized":"2024-03-10T00:52:37.662545+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-10T00:51:52.085771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-10T00:51:52.085771+0000","last_clean_scrub_stamp":"2024-03-10T00:51:52.085771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T08:51:49.918037+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"67'22","reported_seq":64,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-10T00:52:38.021966+0000","last_change":"2024-03-10T00:51:55.118486+0000","last_active":"2024-03-10T00:52:38.021966+0000","last_peered":"2024-03-10T00:52:38.021966+0000","last_clean":"2024-03-10T00:52:38.021966+0000","last_became_active":"2024-03-10T00:51:55.118211+0000","last_became_peered":"2024-03-10T00:51:55.118211+0000","last_unstale":"2024-03-10T00:52:38.021966+0000","last_undegraded":"2024-03-10T00:52:38.021966+0000","last_fullsized":"2024-03-10T00:52:38.021966+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-10T00:51:54.096929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-10T00:51:54.096929+0000","last_clean_scrub_stamp":"2024-03-10T00:51:54.096929+0000","objects_scrubbed":0,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T10:36:53.374217+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":33,"num_read_kb":26,"num_write":22,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":17,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-10T00:52:38.021230+0000","last_change":"2024-03-10T00:51:59.135792+0000","last_active":"2024-03-10T00:52:38.021230+0000","last_peered":"2024-03-10T00:52:38.021230+0000","last_clean":"2024-03-10T00:52:38.021230+0000","last_became_active":"2024-03-10T00:51:59.135512+0000","last_became_peered":"2024-03-10T00:51:59.135512+0000","last_unstale":"2024-03-10T00:52:38.021230+0000","last_undegraded":"2024-03-10T00:52:38.021230+0000","last_fullsized":"2024-03-10T00:52:38.021230+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-10T00:51:58.121476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-10T00:51:58.121476+0000","last_clean_scrub_stamp":"2024-03-10T00:51:58.121476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T10:32:36.652502+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.14","version":"67'11","reported_seq":54,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-10T00:53:18.174594+0000","last_change":"2024-03-10T00:51:57.140078+0000","last_active":"2024-03-10T00:53:18.174594+0000","last_peered":"2024-03-10T00:53:18.174594+0000","last_clean":"2024-03-10T00:53:18.174594+0000","last_became_active":"2024-03-10T00:51:57.139372+0000","last_became_peered":"2024-03-10T00:51:57.139372+0000","last_unstale":"2024-03-10T00:53:18.174594+0000","last_undegraded":"2024-03-10T00:53:18.174594+0000","last_fullsized":"2024-03-10T00:53:18.174594+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-10T00:51:56.116216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-10T00:51:56.116216+0000","last_clean_scrub_stamp":"2024-03-10T00:51:56.116216+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T01:38:48.451295+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":29,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-10T00:52:38.020456+0000","last_change":"2024-03-10T00:51:53.109532+0000","last_active":"2024-03-10T00:52:38.020456+0000","last_peered":"2024-03-10T00:52:38.020456+0000","last_clean":"2024-03-10T00:52:38.020456+0000","last_became_active":"2024-03-10T00:51:53.109212+0000","last_became_peered":"2024-03-10T00:51:53.109212+0000","last_unstale":"2024-03-10T00:52:38.020456+0000","last_undegraded":"2024-03-10T00:52:38.020456+0000","last_fullsized":"2024-03-10T00:52:38.020456+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-10T00:51:52.085771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-10T00:51:52.085771+0000","last_clean_scrub_stamp":"2024-03-10T00:51:52.085771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T11:01:37.331766+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"67'15","reported_seq":53,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-10T00:52:37.553488+0000","last_change":"2024-03-10T00:51:55.117721+0000","last_active":"2024-03-10T00:52:37.553488+0000","last_peered":"2024-03-10T00:52:37.553488+0000","last_clean":"2024-03-10T00:52:37.553488+0000","last_became_active":"2024-03-10T00:51:55.117443+0000","last_became_peered":"2024-03-10T00:51:55.117443+0000","last_unstale":"2024-03-10T00:52:37.553488+0000","last_undegraded":"2024-03-10T00:52:37.553488+0000","last_fullsized":"2024-03-10T00:52:37.553488+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-10T00:51:54.096929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-10T00:51:54.096929+0000","last_clean_scrub_stamp":"2024-03-10T00:51:54.096929+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T12:18:05.172528+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":17,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-10T00:52:37.555605+0000","last_change":"2024-03-10T00:51:59.137447+0000","last_active":"2024-03-10T00:52:37.555605+0000","last_peered":"2024-03-10T00:52:37.555605+0000","last_clean":"2024-03-10T00:52:37.555605+0000","last_became_active":"2024-03-10T00:51:59.137127+0000","last_became_peered":"2024-03-10T00:51:59.137127+0000","last_unstale":"2024-03-10T00:52:37.555605+0000","last_undegraded":"2024-03-10T00:52:37.555605+0000","last_fullsized":"2024-03-10T00:52:37.555605+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-10T00:51:58.121476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-10T00:51:58.121476+0000","last_clean_scrub_stamp":"2024-03-10T00:51:58.121476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T10:40:58.913729+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":21,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-10T00:52:38.022908+0000","last_change":"2024-03-10T00:51:57.139514+0000","last_active":"2024-03-10T00:52:38.022908+0000","last_peered":"2024-03-10T00:52:38.022908+0000","last_clean":"2024-03-10T00:52:38.022908+0000","last_became_active":"2024-03-10T00:51:57.139152+0000","last_became_peered":"2024-03-10T00:51:57.139152+0000","last_unstale":"2024-03-10T00:52:38.022908+0000","last_undegraded":"2024-03-10T00:52:38.022908+0000","last_fullsized":"2024-03-10T00:52:38.022908+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-10T00:51:56.116216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-10T00:51:56.116216+0000","last_clean_scrub_stamp":"2024-03-10T00:51:56.116216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T03:30:30.034401+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":29,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-10T00:52:37.662416+0000","last_change":"2024-03-10T00:51:53.114674+0000","last_active":"2024-03-10T00:52:37.662416+0000","last_peered":"2024-03-10T00:52:37.662416+0000","last_clean":"2024-03-10T00:52:37.662416+0000","last_became_active":"2024-03-10T00:51:53.114376+0000","last_became_peered":"2024-03-10T00:51:53.114376+0000","last_unstale":"2024-03-10T00:52:37.662416+0000","last_undegraded":"2024-03-10T00:52:37.662416+0000","last_fullsized":"2024-03-10T00:52:37.662416+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-10T00:51:52.085771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-10T00:51:52.085771+0000","last_clean_scrub_stamp":"2024-03-10T00:51:52.085771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T08:15:55.103009+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"67'21","reported_seq":65,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-10T00:52:38.021720+0000","last_change":"2024-03-10T00:51:55.109287+0000","last_active":"2024-03-10T00:52:38.021720+0000","last_peered":"2024-03-10T00:52:38.021720+0000","last_clean":"2024-03-10T00:52:38.021720+0000","last_became_active":"2024-03-10T00:51:55.108849+0000","last_became_peered":"2024-03-10T00:51:55.108849+0000","last_unstale":"2024-03-10T00:52:38.021720+0000","last_undegraded":"2024-03-10T00:52:38.021720+0000","last_fullsized":"2024-03-10T00:52:38.021720+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-10T00:51:54.096929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-10T00:51:54.096929+0000","last_clean_scrub_stamp":"2024-03-10T00:51:54.096929+0000","objects_scrubbed":0,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T07:36:31.420197+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":35,"num_read_kb":27,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":17,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-10T00:52:37.662521+0000","last_change":"2024-03-10T00:51:59.266438+0000","last_active":"2024-03-10T00:52:37.662521+0000","last_peered":"2024-03-10T00:52:37.662521+0000","last_clean":"2024-03-10T00:52:37.662521+0000","last_became_active":"2024-03-10T00:51:59.266149+0000","last_became_peered":"2024-03-10T00:51:59.266149+0000","last_unstale":"2024-03-10T00:52:37.662521+0000","last_undegraded":"2024-03-10T00:52:37.662521+0000","last_fullsized":"2024-03-10T00:52:37.662521+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-10T00:51:58.121476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-10T00:51:58.121476+0000","last_clean_scrub_stamp":"2024-03-10T00:51:58.121476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T07:55:15.921266+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":21,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-10T00:52:37.552599+0000","last_change":"2024-03-10T00:51:57.137507+0000","last_active":"2024-03-10T00:52:37.552599+0000","last_peered":"2024-03-10T00:52:37.552599+0000","last_clean":"2024-03-10T00:52:37.552599+0000","last_became_active":"2024-03-10T00:51:57.137235+0000","last_became_peered":"2024-03-10T00:51:57.137235+0000","last_unstale":"2024-03-10T00:52:37.552599+0000","last_undegraded":"2024-03-10T00:52:37.552599+0000","last_fullsized":"2024-03-10T00:52:37.552599+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-10T00:51:56.116216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-10T00:51:56.116216+0000","last_clean_scrub_stamp":"2024-03-10T00:51:56.116216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T02:46:05.371162+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"60'1","reported_seq":37,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-10T00:52:38.019805+0000","last_change":"2024-03-10T00:51:53.114271+0000","last_active":"2024-03-10T00:52:38.019805+0000","last_peered":"2024-03-10T00:52:38.019805+0000","last_clean":"2024-03-10T00:52:38.019805+0000","last_became_active":"2024-03-10T00:51:53.113912+0000","last_became_peered":"2024-03-10T00:51:53.113912+0000","last_unstale":"2024-03-10T00:52:38.019805+0000","last_undegraded":"2024-03-10T00:52:38.019805+0000","last_fullsized":"2024-03-10T00:52:38.019805+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-10T00:51:52.085771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-10T00:51:52.085771+0000","last_clean_scrub_stamp":"2024-03-10T00:51:52.085771+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T02:54:02.668765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":398,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"67'12","reported_seq":46,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-10T00:52:38.022828+0000","last_change":"2024-03-10T00:51:55.119659+0000","last_active":"2024-03-10T00:52:38.022828+0000","last_peered":"2024-03-10T00:52:38.022828+0000","last_clean":"2024-03-10T00:52:38.022828+0000","last_became_active":"2024-03-10T00:51:55.119398+0000","last_became_peered":"2024-03-10T00:51:55.119398+0000","last_unstale":"2024-03-10T00:52:38.022828+0000","last_undegraded":"2024-03-10T00:52:38.022828+0000","last_fullsized":"2024-03-10T00:52:38.022828+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-10T00:51:54.096929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-10T00:51:54.096929+0000","last_clean_scrub_stamp":"2024-03-10T00:51:54.096929+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T06:19:43.507953+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":18,"num_read_kb":14,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":17,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-10T00:52:37.552490+0000","last_change":"2024-03-10T00:51:59.263745+0000","last_active":"2024-03-10T00:52:37.552490+0000","last_peered":"2024-03-10T00:52:37.552490+0000","last_clean":"2024-03-10T00:52:37.552490+0000","last_became_active":"2024-03-10T00:51:59.263366+0000","last_became_peered":"2024-03-10T00:51:59.263366+0000","last_unstale":"2024-03-10T00:52:37.552490+0000","last_undegraded":"2024-03-10T00:52:37.552490+0000","last_fullsized":"2024-03-10T00:52:37.552490+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-10T00:51:58.121476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-10T00:51:58.121476+0000","last_clean_scrub_stamp":"2024-03-10T00:51:58.121476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T04:04:34.367832+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"60'2","reported_seq":45,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-10T00:52:37.662894+0000","last_change":"2024-03-10T00:51:53.119511+0000","last_active":"2024-03-10T00:52:37.662894+0000","last_peered":"2024-03-10T00:52:37.662894+0000","last_clean":"2024-03-10T00:52:37.662894+0000","last_became_active":"2024-03-10T00:51:53.119222+0000","last_became_peered":"2024-03-10T00:51:53.119222+0000","last_unstale":"2024-03-10T00:52:37.662894+0000","last_undegraded":"2024-03-10T00:52:37.662894+0000","last_fullsized":"2024-03-10T00:52:37.662894+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-10T00:51:52.085771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-10T00:51:52.085771+0000","last_clean_scrub_stamp":"2024-03-10T00:51:52.085771+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T06:35:44.779038+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":14,"num_read_kb":14,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"67'27","reported_seq":74,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-10T00:52:37.551960+0000","last_change":"2024-03-10T00:51:55.114646+0000","last_active":"2024-03-10T00:52:37.551960+0000","last_peered":"2024-03-10T00:52:37.551960+0000","last_clean":"2024-03-10T00:52:37.551960+0000","last_became_active":"2024-03-10T00:51:55.114342+0000","last_became_peered":"2024-03-10T00:51:55.114342+0000","last_unstale":"2024-03-10T00:52:37.551960+0000","last_undegraded":"2024-03-10T00:52:37.551960+0000","last_fullsized":"2024-03-10T00:52:37.551960+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-10T00:51:54.096929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-10T00:51:54.096929+0000","last_clean_scrub_stamp":"2024-03-10T00:51:54.096929+0000","objects_scrubbed":0,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T11:30:26.324585+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":44,"num_read_kb":33,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"67'11","reported_seq":51,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-10T00:53:18.174751+0000","last_change":"2024-03-10T00:51:57.143030+0000","last_active":"2024-03-10T00:53:18.174751+0000","last_peered":"2024-03-10T00:53:18.174751+0000","last_clean":"2024-03-10T00:53:18.174751+0000","last_became_active":"2024-03-10T00:51:57.142653+0000","last_became_peered":"2024-03-10T00:51:57.142653+0000","last_unstale":"2024-03-10T00:53:18.174751+0000","last_undegraded":"2024-03-10T00:53:18.174751+0000","last_fullsized":"2024-03-10T00:53:18.174751+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-10T00:51:56.116216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-10T00:51:56.116216+0000","last_clean_scrub_stamp":"2024-03-10T00:51:56.116216+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T05:14:26.367968+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":17,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-10T00:52:38.021258+0000","last_change":"2024-03-10T00:51:59.146184+0000","last_active":"2024-03-10T00:52:38.021258+0000","last_peered":"2024-03-10T00:52:38.021258+0000","last_clean":"2024-03-10T00:52:38.021258+0000","last_became_active":"2024-03-10T00:51:59.145644+0000","last_became_peered":"2024-03-10T00:51:59.145644+0000","last_unstale":"2024-03-10T00:52:38.021258+0000","last_undegraded":"2024-03-10T00:52:38.021258+0000","last_fullsized":"2024-03-10T00:52:38.021258+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-10T00:51:58.121476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-10T00:51:58.121476+0000","last_clean_scrub_stamp":"2024-03-10T00:51:58.121476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T00:54:03.007401+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":29,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-10T00:52:37.662716+0000","last_change":"2024-03-10T00:51:53.118823+0000","last_active":"2024-03-10T00:52:37.662716+0000","last_peered":"2024-03-10T00:52:37.662716+0000","last_clean":"2024-03-10T00:52:37.662716+0000","last_became_active":"2024-03-10T00:51:53.118546+0000","last_became_peered":"2024-03-10T00:51:53.118546+0000","last_unstale":"2024-03-10T00:52:37.662716+0000","last_undegraded":"2024-03-10T00:52:37.662716+0000","last_fullsized":"2024-03-10T00:52:37.662716+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-10T00:51:52.085771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-10T00:51:52.085771+0000","last_clean_scrub_stamp":"2024-03-10T00:51:52.085771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T04:51:14.535355+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"67'24","reported_seq":72,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-10T00:52:38.020372+0000","last_change":"2024-03-10T00:51:55.119156+0000","last_active":"2024-03-10T00:52:38.020372+0000","last_peered":"2024-03-10T00:52:38.020372+0000","last_clean":"2024-03-10T00:52:38.020372+0000","last_became_active":"2024-03-10T00:51:55.118871+0000","last_became_peered":"2024-03-10T00:51:55.118871+0000","last_unstale":"2024-03-10T00:52:38.020372+0000","last_undegraded":"2024-03-10T00:52:38.020372+0000","last_fullsized":"2024-03-10T00:52:38.020372+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-10T00:51:54.096929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-10T00:51:54.096929+0000","last_clean_scrub_stamp":"2024-03-10T00:51:54.096929+0000","objects_scrubbed":0,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T06:07:46.272776+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":32,"num_write":26,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":21,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-10T00:52:37.555488+0000","last_change":"2024-03-10T00:51:57.136934+0000","last_active":"2024-03-10T00:52:37.555488+0000","last_peered":"2024-03-10T00:52:37.555488+0000","last_clean":"2024-03-10T00:52:37.555488+0000","last_became_active":"2024-03-10T00:51:57.136579+0000","last_became_peered":"2024-03-10T00:51:57.136579+0000","last_unstale":"2024-03-10T00:52:37.555488+0000","last_undegraded":"2024-03-10T00:52:37.555488+0000","last_fullsized":"2024-03-10T00:52:37.555488+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-10T00:51:56.116216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-10T00:51:56.116216+0000","last_clean_scrub_stamp":"2024-03-10T00:51:56.116216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T07:56:33.320243+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":17,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-10T00:52:37.662180+0000","last_change":"2024-03-10T00:51:59.137262+0000","last_active":"2024-03-10T00:52:37.662180+0000","last_peered":"2024-03-10T00:52:37.662180+0000","last_clean":"2024-03-10T00:52:37.662180+0000","last_became_active":"2024-03-10T00:51:59.136997+0000","last_became_peered":"2024-03-10T00:51:59.136997+0000","last_unstale":"2024-03-10T00:52:37.662180+0000","last_undegraded":"2024-03-10T00:52:37.662180+0000","last_fullsized":"2024-03-10T00:52:37.662180+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-10T00:51:58.121476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-10T00:51:58.121476+0000","last_clean_scrub_stamp":"2024-03-10T00:51:58.121476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T05:23:57.386618+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":21,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-10T00:52:37.554790+0000","last_change":"2024-03-10T00:51:57.135617+0000","last_active":"2024-03-10T00:52:37.554790+0000","last_peered":"2024-03-10T00:52:37.554790+0000","last_clean":"2024-03-10T00:52:37.554790+0000","last_became_active":"2024-03-10T00:51:57.135026+0000","last_became_peered":"2024-03-10T00:51:57.135026+0000","last_unstale":"2024-03-10T00:52:37.554790+0000","last_undegraded":"2024-03-10T00:52:37.554790+0000","last_fullsized":"2024-03-10T00:52:37.554790+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-10T00:51:56.116216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-10T00:51:56.116216+0000","last_clean_scrub_stamp":"2024-03-10T00:51:56.116216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T08:27:10.673504+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":29,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-10T00:52:37.662875+0000","last_change":"2024-03-10T00:51:53.112002+0000","last_active":"2024-03-10T00:52:37.662875+0000","last_peered":"2024-03-10T00:52:37.662875+0000","last_clean":"2024-03-10T00:52:37.662875+0000","last_became_active":"2024-03-10T00:51:53.111707+0000","last_became_peered":"2024-03-10T00:51:53.111707+0000","last_unstale":"2024-03-10T00:52:37.662875+0000","last_undegraded":"2024-03-10T00:52:37.662875+0000","last_fullsized":"2024-03-10T00:52:37.662875+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-10T00:51:52.085771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-10T00:51:52.085771+0000","last_clean_scrub_stamp":"2024-03-10T00:51:52.085771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T11:33:18.020874+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"67'25","reported_seq":68,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-10T00:52:38.020557+0000","last_change":"2024-03-10T00:51:55.120348+0000","last_active":"2024-03-10T00:52:38.020557+0000","last_peered":"2024-03-10T00:52:38.020557+0000","last_clean":"2024-03-10T00:52:38.020557+0000","last_became_active":"2024-03-10T00:51:55.120009+0000","last_became_peered":"2024-03-10T00:51:55.120009+0000","last_unstale":"2024-03-10T00:52:38.020557+0000","last_undegraded":"2024-03-10T00:52:38.020557+0000","last_fullsized":"2024-03-10T00:52:38.020557+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-10T00:51:54.096929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-10T00:51:54.096929+0000","last_clean_scrub_stamp":"2024-03-10T00:51:54.096929+0000","objects_scrubbed":0,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T09:13:02.827451+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":41,"num_read_kb":29,"num_write":26,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":17,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-10T00:52:38.019995+0000","last_change":"2024-03-10T00:51:59.144510+0000","last_active":"2024-03-10T00:52:38.019995+0000","last_peered":"2024-03-10T00:52:38.019995+0000","last_clean":"2024-03-10T00:52:38.019995+0000","last_became_active":"2024-03-10T00:51:59.144023+0000","last_became_peered":"2024-03-10T00:51:59.144023+0000","last_unstale":"2024-03-10T00:52:38.019995+0000","last_undegraded":"2024-03-10T00:52:38.019995+0000","last_fullsized":"2024-03-10T00:52:38.019995+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-10T00:51:58.121476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-10T00:51:58.121476+0000","last_clean_scrub_stamp":"2024-03-10T00:51:58.121476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T01:27:36.953468+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":21,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-10T00:52:37.554826+0000","last_change":"2024-03-10T00:51:57.134250+0000","last_active":"2024-03-10T00:52:37.554826+0000","last_peered":"2024-03-10T00:52:37.554826+0000","last_clean":"2024-03-10T00:52:37.554826+0000","last_became_active":"2024-03-10T00:51:57.133948+0000","last_became_peered":"2024-03-10T00:51:57.133948+0000","last_unstale":"2024-03-10T00:52:37.554826+0000","last_undegraded":"2024-03-10T00:52:37.554826+0000","last_fullsized":"2024-03-10T00:52:37.554826+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-10T00:51:56.116216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-10T00:51:56.116216+0000","last_clean_scrub_stamp":"2024-03-10T00:51:56.116216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T06:34:24.152549+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":29,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-10T00:52:37.552488+0000","last_change":"2024-03-10T00:51:53.118037+0000","last_active":"2024-03-10T00:52:37.552488+0000","last_peered":"2024-03-10T00:52:37.552488+0000","last_clean":"2024-03-10T00:52:37.552488+0000","last_became_active":"2024-03-10T00:51:53.117612+0000","last_became_peered":"2024-03-10T00:51:53.117612+0000","last_unstale":"2024-03-10T00:52:37.552488+0000","last_undegraded":"2024-03-10T00:52:37.552488+0000","last_fullsized":"2024-03-10T00:52:37.552488+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-10T00:51:52.085771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-10T00:51:52.085771+0000","last_clean_scrub_stamp":"2024-03-10T00:51:52.085771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T08:40:54.041208+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"67'15","reported_seq":53,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-10T00:52:38.022247+0000","last_change":"2024-03-10T00:51:55.119392+0000","last_active":"2024-03-10T00:52:38.022247+0000","last_peered":"2024-03-10T00:52:38.022247+0000","last_clean":"2024-03-10T00:52:38.022247+0000","last_became_active":"2024-03-10T00:51:55.119073+0000","last_became_peered":"2024-03-10T00:51:55.119073+0000","last_unstale":"2024-03-10T00:52:38.022247+0000","last_undegraded":"2024-03-10T00:52:38.022247+0000","last_fullsized":"2024-03-10T00:52:38.022247+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-10T00:51:54.096929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-10T00:51:54.096929+0000","last_clean_scrub_stamp":"2024-03-10T00:51:54.096929+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T09:09:22.672882+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":17,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-10T00:52:38.022308+0000","last_change":"2024-03-10T00:51:59.137543+0000","last_active":"2024-03-10T00:52:38.022308+0000","last_peered":"2024-03-10T00:52:38.022308+0000","last_clean":"2024-03-10T00:52:38.022308+0000","last_became_active":"2024-03-10T00:51:59.137259+0000","last_became_peered":"2024-03-10T00:51:59.137259+0000","last_unstale":"2024-03-10T00:52:38.022308+0000","last_undegraded":"2024-03-10T00:52:38.022308+0000","last_fullsized":"2024-03-10T00:52:38.022308+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-10T00:51:58.121476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-10T00:51:58.121476+0000","last_clean_scrub_stamp":"2024-03-10T00:51:58.121476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T04:45:53.316568+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"64'2","reported_seq":38,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-10T00:52:37.552124+0000","last_change":"2024-03-10T00:51:57.132946+0000","last_active":"2024-03-10T00:52:37.552124+0000","last_peered":"2024-03-10T00:52:37.552124+0000","last_clean":"2024-03-10T00:52:37.552124+0000","last_became_active":"2024-03-10T00:51:53.112432+0000","last_became_peered":"2024-03-10T00:51:53.112432+0000","last_unstale":"2024-03-10T00:52:37.552124+0000","last_undegraded":"2024-03-10T00:52:37.552124+0000","last_fullsized":"2024-03-10T00:52:37.552124+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-10T00:51:52.085771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-10T00:51:52.085771+0000","last_clean_scrub_stamp":"2024-03-10T00:51:52.085771+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T10:49:28.839398+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0015561780000000001,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":29,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-10T00:52:38.020741+0000","last_change":"2024-03-10T00:51:53.115959+0000","last_active":"2024-03-10T00:52:38.020741+0000","last_peered":"2024-03-10T00:52:38.020741+0000","last_clean":"2024-03-10T00:52:38.020741+0000","last_became_active":"2024-03-10T00:51:53.115681+0000","last_became_peered":"2024-03-10T00:51:53.115681+0000","last_unstale":"2024-03-10T00:52:38.020741+0000","last_undegraded":"2024-03-10T00:52:38.020741+0000","last_fullsized":"2024-03-10T00:52:38.020741+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-10T00:51:52.085771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-10T00:51:52.085771+0000","last_clean_scrub_stamp":"2024-03-10T00:51:52.085771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T12:48:38.872556+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.4","version":"67'36","reported_seq":107,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-10T00:53:18.174476+0000","last_change":"2024-03-10T00:51:55.123960+0000","last_active":"2024-03-10T00:53:18.174476+0000","last_peered":"2024-03-10T00:53:18.174476+0000","last_clean":"2024-03-10T00:53:18.174476+0000","last_became_active":"2024-03-10T00:51:55.123642+0000","last_became_peered":"2024-03-10T00:51:55.123642+0000","last_unstale":"2024-03-10T00:53:18.174476+0000","last_undegraded":"2024-03-10T00:53:18.174476+0000","last_fullsized":"2024-03-10T00:53:18.174476+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-10T00:51:54.096929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-10T00:51:54.096929+0000","last_clean_scrub_stamp":"2024-03-10T00:51:54.096929+0000","objects_scrubbed":0,"log_size":36,"log_dups_size":0,"ondisk_log_size":36,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T12:12:52.388286+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":60,"num_read_kb":44,"num_write":32,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":21,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-10T00:52:38.020475+0000","last_change":"2024-03-10T00:51:57.136454+0000","last_active":"2024-03-10T00:52:38.020475+0000","last_peered":"2024-03-10T00:52:38.020475+0000","last_clean":"2024-03-10T00:52:38.020475+0000","last_became_active":"2024-03-10T00:51:57.136014+0000","last_became_peered":"2024-03-10T00:51:57.136014+0000","last_unstale":"2024-03-10T00:52:38.020475+0000","last_undegraded":"2024-03-10T00:52:38.020475+0000","last_fullsized":"2024-03-10T00:52:38.020475+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-10T00:51:56.116216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-10T00:51:56.116216+0000","last_clean_scrub_stamp":"2024-03-10T00:51:56.116216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T11:48:07.545530+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":17,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-10T00:52:37.662976+0000","last_change":"2024-03-10T00:51:59.265490+0000","last_active":"2024-03-10T00:52:37.662976+0000","last_peered":"2024-03-10T00:52:37.662976+0000","last_clean":"2024-03-10T00:52:37.662976+0000","last_became_active":"2024-03-10T00:51:59.265155+0000","last_became_peered":"2024-03-10T00:51:59.265155+0000","last_unstale":"2024-03-10T00:52:37.662976+0000","last_undegraded":"2024-03-10T00:52:37.662976+0000","last_fullsized":"2024-03-10T00:52:37.662976+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-10T00:51:58.121476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-10T00:51:58.121476+0000","last_clean_scrub_stamp":"2024-03-10T00:51:58.121476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T08:51:31.161077+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":29,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-10T00:52:37.551063+0000","last_change":"2024-03-10T00:51:53.111526+0000","last_active":"2024-03-10T00:52:37.551063+0000","last_peered":"2024-03-10T00:52:37.551063+0000","last_clean":"2024-03-10T00:52:37.551063+0000","last_became_active":"2024-03-10T00:51:53.111103+0000","last_became_peered":"2024-03-10T00:51:53.111103+0000","last_unstale":"2024-03-10T00:52:37.551063+0000","last_undegraded":"2024-03-10T00:52:37.551063+0000","last_fullsized":"2024-03-10T00:52:37.551063+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-10T00:51:52.085771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-10T00:51:52.085771+0000","last_clean_scrub_stamp":"2024-03-10T00:51:52.085771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T05:26:32.767017+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":36,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-10T00:52:37.555661+0000","last_change":"2024-03-10T00:51:57.124791+0000","last_active":"2024-03-10T00:52:37.555661+0000","last_peered":"2024-03-10T00:52:37.555661+0000","last_clean":"2024-03-10T00:52:37.555661+0000","last_became_active":"2024-03-10T00:51:53.112252+0000","last_became_peered":"2024-03-10T00:51:53.112252+0000","last_unstale":"2024-03-10T00:52:37.555661+0000","last_undegraded":"2024-03-10T00:52:37.555661+0000","last_fullsized":"2024-03-10T00:52:37.555661+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-10T00:51:52.085771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-10T00:51:52.085771+0000","last_clean_scrub_stamp":"2024-03-10T00:51:52.085771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T09:14:05.516707+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00067091600000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"67'19","reported_seq":64,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-10T00:52:37.553069+0000","last_change":"2024-03-10T00:51:55.118163+0000","last_active":"2024-03-10T00:52:37.553069+0000","last_peered":"2024-03-10T00:52:37.553069+0000","last_clean":"2024-03-10T00:52:37.553069+0000","last_became_active":"2024-03-10T00:51:55.117829+0000","last_became_peered":"2024-03-10T00:51:55.117829+0000","last_unstale":"2024-03-10T00:52:37.553069+0000","last_undegraded":"2024-03-10T00:52:37.553069+0000","last_fullsized":"2024-03-10T00:52:37.553069+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-10T00:51:54.096929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-10T00:51:54.096929+0000","last_clean_scrub_stamp":"2024-03-10T00:51:54.096929+0000","objects_scrubbed":0,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T08:45:59.775240+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":39,"num_read_kb":27,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":21,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-10T00:52:37.554923+0000","last_change":"2024-03-10T00:51:57.138045+0000","last_active":"2024-03-10T00:52:37.554923+0000","last_peered":"2024-03-10T00:52:37.554923+0000","last_clean":"2024-03-10T00:52:37.554923+0000","last_became_active":"2024-03-10T00:51:57.136939+0000","last_became_peered":"2024-03-10T00:51:57.136939+0000","last_unstale":"2024-03-10T00:52:37.554923+0000","last_undegraded":"2024-03-10T00:52:37.554923+0000","last_fullsized":"2024-03-10T00:52:37.554923+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-10T00:51:56.116216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-10T00:51:56.116216+0000","last_clean_scrub_stamp":"2024-03-10T00:51:56.116216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T06:34:37.358733+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":17,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-10T00:52:37.552739+0000","last_change":"2024-03-10T00:51:59.141162+0000","last_active":"2024-03-10T00:52:37.552739+0000","last_peered":"2024-03-10T00:52:37.552739+0000","last_clean":"2024-03-10T00:52:37.552739+0000","last_became_active":"2024-03-10T00:51:59.140795+0000","last_became_peered":"2024-03-10T00:51:59.140795+0000","last_unstale":"2024-03-10T00:52:37.552739+0000","last_undegraded":"2024-03-10T00:52:37.552739+0000","last_fullsized":"2024-03-10T00:52:37.552739+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-10T00:51:58.121476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-10T00:51:58.121476+0000","last_clean_scrub_stamp":"2024-03-10T00:51:58.121476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T09:31:17.571248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"69'188","reported_seq":341,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-10T00:52:39.656783+0000","last_change":"2024-03-10T00:51:41.219917+0000","last_active":"2024-03-10T00:52:39.656783+0000","last_peered":"2024-03-10T00:52:39.656783+0000","last_clean":"2024-03-10T00:52:39.656783+0000","last_became_active":"2024-03-10T00:51:41.219507+0000","last_became_peered":"2024-03-10T00:51:41.219507+0000","last_unstale":"2024-03-10T00:52:39.656783+0000","last_undegraded":"2024-03-10T00:52:39.656783+0000","last_fullsized":"2024-03-10T00:52:39.656783+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-10T00:49:25.306169+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-10T00:49:25.306169+0000","last_clean_scrub_stamp":"2024-03-10T00:49:25.306169+0000","objects_scrubbed":0,"log_size":188,"log_dups_size":0,"ondisk_log_size":188,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T12:12:42.766777+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":242,"num_read_kb":333,"num_write":245,"num_write_kb":3982,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":29,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-10T00:52:38.020996+0000","last_change":"2024-03-10T00:51:53.115998+0000","last_active":"2024-03-10T00:52:38.020996+0000","last_peered":"2024-03-10T00:52:38.020996+0000","last_clean":"2024-03-10T00:52:38.020996+0000","last_became_active":"2024-03-10T00:51:53.115668+0000","last_became_peered":"2024-03-10T00:51:53.115668+0000","last_unstale":"2024-03-10T00:52:38.020996+0000","last_undegraded":"2024-03-10T00:52:38.020996+0000","last_fullsized":"2024-03-10T00:52:38.020996+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-10T00:51:52.085771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-10T00:51:52.085771+0000","last_clean_scrub_stamp":"2024-03-10T00:51:52.085771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T09:59:09.605361+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"67'16","reported_seq":66,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-10T00:53:18.174735+0000","last_change":"2024-03-10T00:51:55.115838+0000","last_active":"2024-03-10T00:53:18.174735+0000","last_peered":"2024-03-10T00:53:18.174735+0000","last_clean":"2024-03-10T00:53:18.174735+0000","last_became_active":"2024-03-10T00:51:55.115529+0000","last_became_peered":"2024-03-10T00:51:55.115529+0000","last_unstale":"2024-03-10T00:53:18.174735+0000","last_undegraded":"2024-03-10T00:53:18.174735+0000","last_fullsized":"2024-03-10T00:53:18.174735+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-10T00:51:54.096929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-10T00:51:54.096929+0000","last_clean_scrub_stamp":"2024-03-10T00:51:54.096929+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T12:11:12.704417+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":25,"num_read_kb":15,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":21,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-10T00:52:37.662703+0000","last_change":"2024-03-10T00:51:57.141083+0000","last_active":"2024-03-10T00:52:37.662703+0000","last_peered":"2024-03-10T00:52:37.662703+0000","last_clean":"2024-03-10T00:52:37.662703+0000","last_became_active":"2024-03-10T00:51:57.140761+0000","last_became_peered":"2024-03-10T00:51:57.140761+0000","last_unstale":"2024-03-10T00:52:37.662703+0000","last_undegraded":"2024-03-10T00:52:37.662703+0000","last_fullsized":"2024-03-10T00:52:37.662703+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-10T00:51:56.116216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-10T00:51:56.116216+0000","last_clean_scrub_stamp":"2024-03-10T00:51:56.116216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T04:13:15.481320+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":17,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-10T00:52:37.551542+0000","last_change":"2024-03-10T00:51:59.142614+0000","last_active":"2024-03-10T00:52:37.551542+0000","last_peered":"2024-03-10T00:52:37.551542+0000","last_clean":"2024-03-10T00:52:37.551542+0000","last_became_active":"2024-03-10T00:51:59.142332+0000","last_became_peered":"2024-03-10T00:51:59.142332+0000","last_unstale":"2024-03-10T00:52:37.551542+0000","last_undegraded":"2024-03-10T00:52:37.551542+0000","last_fullsized":"2024-03-10T00:52:37.551542+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-10T00:51:58.121476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-10T00:51:58.121476+0000","last_clean_scrub_stamp":"2024-03-10T00:51:58.121476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T11:14:07.678440+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":29,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-10T00:52:38.021152+0000","last_change":"2024-03-10T00:51:53.117913+0000","last_active":"2024-03-10T00:52:38.021152+0000","last_peered":"2024-03-10T00:52:38.021152+0000","last_clean":"2024-03-10T00:52:38.021152+0000","last_became_active":"2024-03-10T00:51:53.117308+0000","last_became_peered":"2024-03-10T00:51:53.117308+0000","last_unstale":"2024-03-10T00:52:38.021152+0000","last_undegraded":"2024-03-10T00:52:38.021152+0000","last_fullsized":"2024-03-10T00:52:38.021152+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-10T00:51:52.085771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-10T00:51:52.085771+0000","last_clean_scrub_stamp":"2024-03-10T00:51:52.085771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T06:30:43.045812+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"67'5","reported_seq":123,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-10T00:53:21.592873+0000","last_change":"2024-03-10T00:51:57.134275+0000","last_active":"2024-03-10T00:53:21.592873+0000","last_peered":"2024-03-10T00:53:21.592873+0000","last_clean":"2024-03-10T00:53:21.592873+0000","last_became_active":"2024-03-10T00:51:53.111611+0000","last_became_peered":"2024-03-10T00:51:53.111611+0000","last_unstale":"2024-03-10T00:53:21.592873+0000","last_undegraded":"2024-03-10T00:53:21.592873+0000","last_fullsized":"2024-03-10T00:53:21.592873+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-10T00:51:52.085771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-10T00:51:52.085771+0000","last_clean_scrub_stamp":"2024-03-10T00:51:52.085771+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T08:49:42.631320+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00098723699999999997,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":84,"num_read_kb":79,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"67'12","reported_seq":43,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-10T00:52:38.021378+0000","last_change":"2024-03-10T00:51:55.119330+0000","last_active":"2024-03-10T00:52:38.021378+0000","last_peered":"2024-03-10T00:52:38.021378+0000","last_clean":"2024-03-10T00:52:38.021378+0000","last_became_active":"2024-03-10T00:51:55.119047+0000","last_became_peered":"2024-03-10T00:51:55.119047+0000","last_unstale":"2024-03-10T00:52:38.021378+0000","last_undegraded":"2024-03-10T00:52:38.021378+0000","last_fullsized":"2024-03-10T00:52:38.021378+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-10T00:51:54.096929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-10T00:51:54.096929+0000","last_clean_scrub_stamp":"2024-03-10T00:51:54.096929+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T04:52:14.508539+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":18,"num_read_kb":12,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":21,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-10T00:52:37.552933+0000","last_change":"2024-03-10T00:51:57.140671+0000","last_active":"2024-03-10T00:52:37.552933+0000","last_peered":"2024-03-10T00:52:37.552933+0000","last_clean":"2024-03-10T00:52:37.552933+0000","last_became_active":"2024-03-10T00:51:57.140186+0000","last_became_peered":"2024-03-10T00:51:57.140186+0000","last_unstale":"2024-03-10T00:52:37.552933+0000","last_undegraded":"2024-03-10T00:52:37.552933+0000","last_fullsized":"2024-03-10T00:52:37.552933+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-10T00:51:56.116216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-10T00:51:56.116216+0000","last_clean_scrub_stamp":"2024-03-10T00:51:56.116216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T09:29:31.087034+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":17,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-10T00:52:37.551016+0000","last_change":"2024-03-10T00:51:59.264751+0000","last_active":"2024-03-10T00:52:37.551016+0000","last_peered":"2024-03-10T00:52:37.551016+0000","last_clean":"2024-03-10T00:52:37.551016+0000","last_became_active":"2024-03-10T00:51:59.264375+0000","last_became_peered":"2024-03-10T00:51:59.264375+0000","last_unstale":"2024-03-10T00:52:37.551016+0000","last_undegraded":"2024-03-10T00:52:37.551016+0000","last_fullsized":"2024-03-10T00:52:37.551016+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-10T00:51:58.121476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-10T00:51:58.121476+0000","last_clean_scrub_stamp":"2024-03-10T00:51:58.121476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T01:20:42.245941+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":29,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-10T00:52:37.551937+0000","last_change":"2024-03-10T00:51:53.110714+0000","last_active":"2024-03-10T00:52:37.551937+0000","last_peered":"2024-03-10T00:52:37.551937+0000","last_clean":"2024-03-10T00:52:37.551937+0000","last_became_active":"2024-03-10T00:51:53.110240+0000","last_became_peered":"2024-03-10T00:51:53.110240+0000","last_unstale":"2024-03-10T00:52:37.551937+0000","last_undegraded":"2024-03-10T00:52:37.551937+0000","last_fullsized":"2024-03-10T00:52:37.551937+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-10T00:51:52.085771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-10T00:51:52.085771+0000","last_clean_scrub_stamp":"2024-03-10T00:51:52.085771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T09:51:21.505184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.3","version":"67'25","reported_seq":73,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-10T00:52:38.022041+0000","last_change":"2024-03-10T00:51:55.115268+0000","last_active":"2024-03-10T00:52:38.022041+0000","last_peered":"2024-03-10T00:52:38.022041+0000","last_clean":"2024-03-10T00:52:38.022041+0000","last_became_active":"2024-03-10T00:51:55.114979+0000","last_became_peered":"2024-03-10T00:51:55.114979+0000","last_unstale":"2024-03-10T00:52:38.022041+0000","last_undegraded":"2024-03-10T00:52:38.022041+0000","last_fullsized":"2024-03-10T00:52:38.022041+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-10T00:51:54.096929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-10T00:51:54.096929+0000","last_clean_scrub_stamp":"2024-03-10T00:51:54.096929+0000","objects_scrubbed":0,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T12:51:17.750576+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":48,"num_read_kb":33,"num_write":28,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":21,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-10T00:52:38.020715+0000","last_change":"2024-03-10T00:51:57.135661+0000","last_active":"2024-03-10T00:52:38.020715+0000","last_peered":"2024-03-10T00:52:38.020715+0000","last_clean":"2024-03-10T00:52:38.020715+0000","last_became_active":"2024-03-10T00:51:57.135317+0000","last_became_peered":"2024-03-10T00:51:57.135317+0000","last_unstale":"2024-03-10T00:52:38.020715+0000","last_undegraded":"2024-03-10T00:52:38.020715+0000","last_fullsized":"2024-03-10T00:52:38.020715+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-10T00:51:56.116216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-10T00:51:56.116216+0000","last_clean_scrub_stamp":"2024-03-10T00:51:56.116216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T07:04:45.110526+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":17,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-10T00:52:38.019786+0000","last_change":"2024-03-10T00:51:59.147816+0000","last_active":"2024-03-10T00:52:38.019786+0000","last_peered":"2024-03-10T00:52:38.019786+0000","last_clean":"2024-03-10T00:52:38.019786+0000","last_became_active":"2024-03-10T00:51:59.147088+0000","last_became_peered":"2024-03-10T00:51:59.147088+0000","last_unstale":"2024-03-10T00:52:38.019786+0000","last_undegraded":"2024-03-10T00:52:38.019786+0000","last_fullsized":"2024-03-10T00:52:38.019786+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-10T00:51:58.121476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-10T00:51:58.121476+0000","last_clean_scrub_stamp":"2024-03-10T00:51:58.121476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T02:44:35.968610+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":29,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-10T00:52:37.553346+0000","last_change":"2024-03-10T00:51:53.117039+0000","last_active":"2024-03-10T00:52:37.553346+0000","last_peered":"2024-03-10T00:52:37.553346+0000","last_clean":"2024-03-10T00:52:37.553346+0000","last_became_active":"2024-03-10T00:51:53.116259+0000","last_became_peered":"2024-03-10T00:51:53.116259+0000","last_unstale":"2024-03-10T00:52:37.553346+0000","last_undegraded":"2024-03-10T00:52:37.553346+0000","last_fullsized":"2024-03-10T00:52:37.553346+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-10T00:51:52.085771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-10T00:51:52.085771+0000","last_clean_scrub_stamp":"2024-03-10T00:51:52.085771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T06:38:54.512952+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.2","version":"67'16","reported_seq":52,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-10T00:52:37.551724+0000","last_change":"2024-03-10T00:51:55.119021+0000","last_active":"2024-03-10T00:52:37.551724+0000","last_peered":"2024-03-10T00:52:37.551724+0000","last_clean":"2024-03-10T00:52:37.551724+0000","last_became_active":"2024-03-10T00:51:55.118661+0000","last_became_peered":"2024-03-10T00:51:55.118661+0000","last_unstale":"2024-03-10T00:52:37.551724+0000","last_undegraded":"2024-03-10T00:52:37.551724+0000","last_fullsized":"2024-03-10T00:52:37.551724+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-10T00:51:54.096929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-10T00:51:54.096929+0000","last_clean_scrub_stamp":"2024-03-10T00:51:54.096929+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T07:19:05.573264+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":18,"num_write":16,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3","version":"67'11","reported_seq":51,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-10T00:53:18.174467+0000","last_change":"2024-03-10T00:51:57.135495+0000","last_active":"2024-03-10T00:53:18.174467+0000","last_peered":"2024-03-10T00:53:18.174467+0000","last_clean":"2024-03-10T00:53:18.174467+0000","last_became_active":"2024-03-10T00:51:57.135156+0000","last_became_peered":"2024-03-10T00:51:57.135156+0000","last_unstale":"2024-03-10T00:53:18.174467+0000","last_undegraded":"2024-03-10T00:53:18.174467+0000","last_fullsized":"2024-03-10T00:53:18.174467+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-10T00:51:56.116216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-10T00:51:56.116216+0000","last_clean_scrub_stamp":"2024-03-10T00:51:56.116216+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T04:31:48.567337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":17,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-10T00:52:37.662841+0000","last_change":"2024-03-10T00:51:59.265807+0000","last_active":"2024-03-10T00:52:37.662841+0000","last_peered":"2024-03-10T00:52:37.662841+0000","last_clean":"2024-03-10T00:52:37.662841+0000","last_became_active":"2024-03-10T00:51:59.265510+0000","last_became_peered":"2024-03-10T00:51:59.265510+0000","last_unstale":"2024-03-10T00:52:37.662841+0000","last_undegraded":"2024-03-10T00:52:37.662841+0000","last_fullsized":"2024-03-10T00:52:37.662841+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-10T00:51:58.121476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-10T00:51:58.121476+0000","last_clean_scrub_stamp":"2024-03-10T00:51:58.121476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T06:39:25.973402+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.6","version":"60'1","reported_seq":30,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-10T00:52:38.021499+0000","last_change":"2024-03-10T00:51:53.117648+0000","last_active":"2024-03-10T00:52:38.021499+0000","last_peered":"2024-03-10T00:52:38.021499+0000","last_clean":"2024-03-10T00:52:38.021499+0000","last_became_active":"2024-03-10T00:51:53.117331+0000","last_became_peered":"2024-03-10T00:51:53.117331+0000","last_unstale":"2024-03-10T00:52:38.021499+0000","last_undegraded":"2024-03-10T00:52:38.021499+0000","last_fullsized":"2024-03-10T00:52:38.021499+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-10T00:51:52.085771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-10T00:51:52.085771+0000","last_clean_scrub_stamp":"2024-03-10T00:51:52.085771+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T08:08:21.440050+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1","version":"67'20","reported_seq":58,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-10T00:52:38.020245+0000","last_change":"2024-03-10T00:51:55.115843+0000","last_active":"2024-03-10T00:52:38.020245+0000","last_peered":"2024-03-10T00:52:38.020245+0000","last_clean":"2024-03-10T00:52:38.020245+0000","last_became_active":"2024-03-10T00:51:55.115549+0000","last_became_peered":"2024-03-10T00:51:55.115549+0000","last_unstale":"2024-03-10T00:52:38.020245+0000","last_undegraded":"2024-03-10T00:52:38.020245+0000","last_fullsized":"2024-03-10T00:52:38.020245+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-10T00:51:54.096929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-10T00:51:54.096929+0000","last_clean_scrub_stamp":"2024-03-10T00:51:54.096929+0000","objects_scrubbed":0,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T06:34:56.720783+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":30,"num_read_kb":22,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.0","version":"67'11","reported_seq":51,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-10T00:53:18.174463+0000","last_change":"2024-03-10T00:51:57.141295+0000","last_active":"2024-03-10T00:53:18.174463+0000","last_peered":"2024-03-10T00:53:18.174463+0000","last_clean":"2024-03-10T00:53:18.174463+0000","last_became_active":"2024-03-10T00:51:57.140997+0000","last_became_peered":"2024-03-10T00:51:57.140997+0000","last_unstale":"2024-03-10T00:53:18.174463+0000","last_undegraded":"2024-03-10T00:53:18.174463+0000","last_fullsized":"2024-03-10T00:53:18.174463+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-10T00:51:56.116216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-10T00:51:56.116216+0000","last_clean_scrub_stamp":"2024-03-10T00:51:56.116216+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T10:25:44.950288+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":17,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-10T00:52:38.020575+0000","last_change":"2024-03-10T00:51:59.135775+0000","last_active":"2024-03-10T00:52:38.020575+0000","last_peered":"2024-03-10T00:52:38.020575+0000","last_clean":"2024-03-10T00:52:38.020575+0000","last_became_active":"2024-03-10T00:51:59.135492+0000","last_became_peered":"2024-03-10T00:51:59.135492+0000","last_unstale":"2024-03-10T00:52:38.020575+0000","last_undegraded":"2024-03-10T00:52:38.020575+0000","last_fullsized":"2024-03-10T00:52:38.020575+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-10T00:51:58.121476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-10T00:51:58.121476+0000","last_clean_scrub_stamp":"2024-03-10T00:51:58.121476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T08:55:04.887116+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"60'1","reported_seq":37,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-10T00:52:38.022769+0000","last_change":"2024-03-10T00:51:53.112758+0000","last_active":"2024-03-10T00:52:38.022769+0000","last_peered":"2024-03-10T00:52:38.022769+0000","last_clean":"2024-03-10T00:52:38.022769+0000","last_became_active":"2024-03-10T00:51:53.111694+0000","last_became_peered":"2024-03-10T00:51:53.111694+0000","last_unstale":"2024-03-10T00:52:38.022769+0000","last_undegraded":"2024-03-10T00:52:38.022769+0000","last_fullsized":"2024-03-10T00:52:38.022769+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-10T00:51:52.085771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-10T00:51:52.085771+0000","last_clean_scrub_stamp":"2024-03-10T00:51:52.085771+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T10:03:50.587703+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":875,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.0","version":"67'32","reported_seq":83,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-10T00:52:38.020372+0000","last_change":"2024-03-10T00:51:55.116821+0000","last_active":"2024-03-10T00:52:38.020372+0000","last_peered":"2024-03-10T00:52:38.020372+0000","last_clean":"2024-03-10T00:52:38.020372+0000","last_became_active":"2024-03-10T00:51:55.116480+0000","last_became_peered":"2024-03-10T00:51:55.116480+0000","last_unstale":"2024-03-10T00:52:38.020372+0000","last_undegraded":"2024-03-10T00:52:38.020372+0000","last_fullsized":"2024-03-10T00:52:38.020372+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-10T00:51:54.096929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-10T00:51:54.096929+0000","last_clean_scrub_stamp":"2024-03-10T00:51:54.096929+0000","objects_scrubbed":0,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T07:27:59.886118+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":52,"num_read_kb":38,"num_write":32,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":21,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-10T00:52:38.020090+0000","last_change":"2024-03-10T00:51:57.135984+0000","last_active":"2024-03-10T00:52:38.020090+0000","last_peered":"2024-03-10T00:52:38.020090+0000","last_clean":"2024-03-10T00:52:38.020090+0000","last_became_active":"2024-03-10T00:51:57.135386+0000","last_became_peered":"2024-03-10T00:51:57.135386+0000","last_unstale":"2024-03-10T00:52:38.020090+0000","last_undegraded":"2024-03-10T00:52:38.020090+0000","last_fullsized":"2024-03-10T00:52:38.020090+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-10T00:51:56.116216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-10T00:51:56.116216+0000","last_clean_scrub_stamp":"2024-03-10T00:51:56.116216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T11:18:38.081117+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":17,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-10T00:52:37.553023+0000","last_change":"2024-03-10T00:51:59.141683+0000","last_active":"2024-03-10T00:52:37.553023+0000","last_peered":"2024-03-10T00:52:37.553023+0000","last_clean":"2024-03-10T00:52:37.553023+0000","last_became_active":"2024-03-10T00:51:59.141437+0000","last_became_peered":"2024-03-10T00:51:59.141437+0000","last_unstale":"2024-03-10T00:52:37.553023+0000","last_undegraded":"2024-03-10T00:52:37.553023+0000","last_fullsized":"2024-03-10T00:52:37.553023+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-10T00:51:58.121476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-10T00:51:58.121476+0000","last_clean_scrub_stamp":"2024-03-10T00:51:58.121476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T10:12:35.650969+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.e","version":"67'11","reported_seq":51,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-10T00:53:18.174707+0000","last_change":"2024-03-10T00:51:57.135230+0000","last_active":"2024-03-10T00:53:18.174707+0000","last_peered":"2024-03-10T00:53:18.174707+0000","last_clean":"2024-03-10T00:53:18.174707+0000","last_became_active":"2024-03-10T00:51:57.134202+0000","last_became_peered":"2024-03-10T00:51:57.134202+0000","last_unstale":"2024-03-10T00:53:18.174707+0000","last_undegraded":"2024-03-10T00:53:18.174707+0000","last_fullsized":"2024-03-10T00:53:18.174707+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-10T00:51:56.116216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-10T00:51:56.116216+0000","last_clean_scrub_stamp":"2024-03-10T00:51:56.116216+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T05:00:15.821525+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":29,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-10T00:52:38.020277+0000","last_change":"2024-03-10T00:51:53.112469+0000","last_active":"2024-03-10T00:52:38.020277+0000","last_peered":"2024-03-10T00:52:38.020277+0000","last_clean":"2024-03-10T00:52:38.020277+0000","last_became_active":"2024-03-10T00:51:53.108351+0000","last_became_peered":"2024-03-10T00:51:53.108351+0000","last_unstale":"2024-03-10T00:52:38.020277+0000","last_undegraded":"2024-03-10T00:52:38.020277+0000","last_fullsized":"2024-03-10T00:52:38.020277+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-10T00:51:52.085771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-10T00:51:52.085771+0000","last_clean_scrub_stamp":"2024-03-10T00:51:52.085771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T09:25:32.877715+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"67'15","reported_seq":50,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-10T00:52:37.552908+0000","last_change":"2024-03-10T00:51:55.124009+0000","last_active":"2024-03-10T00:52:37.552908+0000","last_peered":"2024-03-10T00:52:37.552908+0000","last_clean":"2024-03-10T00:52:37.552908+0000","last_became_active":"2024-03-10T00:51:55.123632+0000","last_became_peered":"2024-03-10T00:51:55.123632+0000","last_unstale":"2024-03-10T00:52:37.552908+0000","last_undegraded":"2024-03-10T00:52:37.552908+0000","last_fullsized":"2024-03-10T00:52:37.552908+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-10T00:51:54.096929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-10T00:51:54.096929+0000","last_clean_scrub_stamp":"2024-03-10T00:51:54.096929+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T11:48:51.675319+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":17,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-10T00:52:38.021814+0000","last_change":"2024-03-10T00:51:59.263882+0000","last_active":"2024-03-10T00:52:38.021814+0000","last_peered":"2024-03-10T00:52:38.021814+0000","last_clean":"2024-03-10T00:52:38.021814+0000","last_became_active":"2024-03-10T00:51:59.263462+0000","last_became_peered":"2024-03-10T00:51:59.263462+0000","last_unstale":"2024-03-10T00:52:38.021814+0000","last_undegraded":"2024-03-10T00:52:38.021814+0000","last_fullsized":"2024-03-10T00:52:38.021814+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-10T00:51:58.121476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-10T00:51:58.121476+0000","last_clean_scrub_stamp":"2024-03-10T00:51:58.121476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T01:44:14.373567+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":21,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-10T00:52:37.552905+0000","last_change":"2024-03-10T00:51:57.140451+0000","last_active":"2024-03-10T00:52:37.552905+0000","last_peered":"2024-03-10T00:52:37.552905+0000","last_clean":"2024-03-10T00:52:37.552905+0000","last_became_active":"2024-03-10T00:51:57.139816+0000","last_became_peered":"2024-03-10T00:51:57.139816+0000","last_unstale":"2024-03-10T00:52:37.552905+0000","last_undegraded":"2024-03-10T00:52:37.552905+0000","last_fullsized":"2024-03-10T00:52:37.552905+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-10T00:51:56.116216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-10T00:51:56.116216+0000","last_clean_scrub_stamp":"2024-03-10T00:51:56.116216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T03:29:46.383843+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":29,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-10T00:52:38.020147+0000","last_change":"2024-03-10T00:51:53.107401+0000","last_active":"2024-03-10T00:52:38.020147+0000","last_peered":"2024-03-10T00:52:38.020147+0000","last_clean":"2024-03-10T00:52:38.020147+0000","last_became_active":"2024-03-10T00:51:53.107121+0000","last_became_peered":"2024-03-10T00:51:53.107121+0000","last_unstale":"2024-03-10T00:52:38.020147+0000","last_undegraded":"2024-03-10T00:52:38.020147+0000","last_fullsized":"2024-03-10T00:52:38.020147+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-10T00:51:52.085771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-10T00:51:52.085771+0000","last_clean_scrub_stamp":"2024-03-10T00:51:52.085771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T03:19:09.033740+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"67'17","reported_seq":56,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-10T00:52:38.021182+0000","last_change":"2024-03-10T00:51:55.119294+0000","last_active":"2024-03-10T00:52:38.021182+0000","last_peered":"2024-03-10T00:52:38.021182+0000","last_clean":"2024-03-10T00:52:38.021182+0000","last_became_active":"2024-03-10T00:51:55.118985+0000","last_became_peered":"2024-03-10T00:51:55.118985+0000","last_unstale":"2024-03-10T00:52:38.021182+0000","last_undegraded":"2024-03-10T00:52:38.021182+0000","last_fullsized":"2024-03-10T00:52:38.021182+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-10T00:51:54.096929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-10T00:51:54.096929+0000","last_clean_scrub_stamp":"2024-03-10T00:51:54.096929+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T04:41:40.524309+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":29,"num_read_kb":21,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":17,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-10T00:52:37.554923+0000","last_change":"2024-03-10T00:51:59.139953+0000","last_active":"2024-03-10T00:52:37.554923+0000","last_peered":"2024-03-10T00:52:37.554923+0000","last_clean":"2024-03-10T00:52:37.554923+0000","last_became_active":"2024-03-10T00:51:59.139571+0000","last_became_peered":"2024-03-10T00:51:59.139571+0000","last_unstale":"2024-03-10T00:52:37.554923+0000","last_undegraded":"2024-03-10T00:52:37.554923+0000","last_fullsized":"2024-03-10T00:52:37.554923+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-10T00:51:58.121476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-10T00:51:58.121476+0000","last_clean_scrub_stamp":"2024-03-10T00:51:58.121476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T05:59:57.853376+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":21,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-10T00:52:37.551970+0000","last_change":"2024-03-10T00:51:57.135459+0000","last_active":"2024-03-10T00:52:37.551970+0000","last_peered":"2024-03-10T00:52:37.551970+0000","last_clean":"2024-03-10T00:52:37.551970+0000","last_became_active":"2024-03-10T00:51:57.134429+0000","last_became_peered":"2024-03-10T00:51:57.134429+0000","last_unstale":"2024-03-10T00:52:37.551970+0000","last_undegraded":"2024-03-10T00:52:37.551970+0000","last_fullsized":"2024-03-10T00:52:37.551970+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-10T00:51:56.116216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-10T00:51:56.116216+0000","last_clean_scrub_stamp":"2024-03-10T00:51:56.116216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T02:29:37.107341+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":29,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-10T00:52:38.019941+0000","last_change":"2024-03-10T00:51:53.119667+0000","last_active":"2024-03-10T00:52:38.019941+0000","last_peered":"2024-03-10T00:52:38.019941+0000","last_clean":"2024-03-10T00:52:38.019941+0000","last_became_active":"2024-03-10T00:51:53.119285+0000","last_became_peered":"2024-03-10T00:51:53.119285+0000","last_unstale":"2024-03-10T00:52:38.019941+0000","last_undegraded":"2024-03-10T00:52:38.019941+0000","last_fullsized":"2024-03-10T00:52:38.019941+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-10T00:51:52.085771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-10T00:51:52.085771+0000","last_clean_scrub_stamp":"2024-03-10T00:51:52.085771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T09:52:08.296964+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.d","version":"67'17","reported_seq":53,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-10T00:52:38.021283+0000","last_change":"2024-03-10T00:51:55.119216+0000","last_active":"2024-03-10T00:52:38.021283+0000","last_peered":"2024-03-10T00:52:38.021283+0000","last_clean":"2024-03-10T00:52:38.021283+0000","last_became_active":"2024-03-10T00:51:55.118921+0000","last_became_peered":"2024-03-10T00:51:55.118921+0000","last_unstale":"2024-03-10T00:52:38.021283+0000","last_undegraded":"2024-03-10T00:52:38.021283+0000","last_fullsized":"2024-03-10T00:52:38.021283+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-10T00:51:54.096929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-10T00:51:54.096929+0000","last_clean_scrub_stamp":"2024-03-10T00:51:54.096929+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T10:38:05.104385+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":19,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":17,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-10T00:52:38.021421+0000","last_change":"2024-03-10T00:51:59.139583+0000","last_active":"2024-03-10T00:52:38.021421+0000","last_peered":"2024-03-10T00:52:38.021421+0000","last_clean":"2024-03-10T00:52:38.021421+0000","last_became_active":"2024-03-10T00:51:59.139298+0000","last_became_peered":"2024-03-10T00:51:59.139298+0000","last_unstale":"2024-03-10T00:52:38.021421+0000","last_undegraded":"2024-03-10T00:52:38.021421+0000","last_fullsized":"2024-03-10T00:52:38.021421+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-10T00:51:58.121476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-10T00:51:58.121476+0000","last_clean_scrub_stamp":"2024-03-10T00:51:58.121476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T09:02:31.591573+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"67'11","reported_seq":51,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-10T00:53:18.174467+0000","last_change":"2024-03-10T00:51:57.136114+0000","last_active":"2024-03-10T00:53:18.174467+0000","last_peered":"2024-03-10T00:53:18.174467+0000","last_clean":"2024-03-10T00:53:18.174467+0000","last_became_active":"2024-03-10T00:51:57.135821+0000","last_became_peered":"2024-03-10T00:51:57.135821+0000","last_unstale":"2024-03-10T00:53:18.174467+0000","last_undegraded":"2024-03-10T00:53:18.174467+0000","last_fullsized":"2024-03-10T00:53:18.174467+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-10T00:51:56.116216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-10T00:51:56.116216+0000","last_clean_scrub_stamp":"2024-03-10T00:51:56.116216+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T01:52:41.238751+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":29,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-10T00:52:38.021547+0000","last_change":"2024-03-10T00:51:53.108796+0000","last_active":"2024-03-10T00:52:38.021547+0000","last_peered":"2024-03-10T00:52:38.021547+0000","last_clean":"2024-03-10T00:52:38.021547+0000","last_became_active":"2024-03-10T00:51:53.108395+0000","last_became_peered":"2024-03-10T00:51:53.108395+0000","last_unstale":"2024-03-10T00:52:38.021547+0000","last_undegraded":"2024-03-10T00:52:38.021547+0000","last_fullsized":"2024-03-10T00:52:38.021547+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-10T00:51:52.085771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-10T00:51:52.085771+0000","last_clean_scrub_stamp":"2024-03-10T00:51:52.085771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T12:14:51.940399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"67'10","reported_seq":40,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-10T00:52:38.020986+0000","last_change":"2024-03-10T00:51:55.117575+0000","last_active":"2024-03-10T00:52:38.020986+0000","last_peered":"2024-03-10T00:52:38.020986+0000","last_clean":"2024-03-10T00:52:38.020986+0000","last_became_active":"2024-03-10T00:51:55.117270+0000","last_became_peered":"2024-03-10T00:51:55.117270+0000","last_unstale":"2024-03-10T00:52:38.020986+0000","last_undegraded":"2024-03-10T00:52:38.020986+0000","last_fullsized":"2024-03-10T00:52:38.020986+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-10T00:51:54.096929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-10T00:51:54.096929+0000","last_clean_scrub_stamp":"2024-03-10T00:51:54.096929+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T08:32:41.786649+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":383,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1149,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":88,"ondisk_log_size":88,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":210,"num_object_clones":0,"num_object_copies":630,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":210,"num_whiteouts":0,"num_read":989,"num_read_kb":714,"num_write":609,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":591,"ondisk_log_size":591,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1457,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":34,"num_read_kb":34,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4371,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":96,"acting":96,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":242,"num_read_kb":333,"num_write":245,"num_write_kb":3982,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":188,"ondisk_log_size":188,"up":3,"acting":3,"num_store_stats":3},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":84,"num_read_kb":79,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":9,"acting":9,"num_store_stats":7}],"osd_stats":[{"osd":7,"up_from":54,"seq":231928234007,"num_pgs":53,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93741056,"kb_used":32520,"kb_used_data":1340,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93708536,"statfs":{"total":95990841344,"available":95957540864,"internally_reserved":0,"allocated":1372160,"data_stored":908965,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":46,"seq":197568495644,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93741056,"kb_used":28336,"kb_used_data":1320,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93712720,"statfs":{"total":95990841344,"available":95961825280,"internally_reserved":0,"allocated":1351680,"data_stored":907436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574903,"num_pgs":57,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93741056,"kb_used":31976,"kb_used_data":796,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93709080,"statfs":{"total":95990841344,"available":95958097920,"internally_reserved":0,"allocated":815104,"data_stored":318442,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738429,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93741056,"kb_used":32524,"kb_used_data":1344,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93708532,"statfs":{"total":95990841344,"available":95957536768,"internally_reserved":0,"allocated":1376256,"data_stored":909092,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411378,"num_pgs":36,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93741056,"kb_used":27748,"kb_used_data":728,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93713308,"statfs":{"total":95990841344,"available":95962427392,"internally_reserved":0,"allocated":745472,"data_stored":316772,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182445,"num_pgs":56,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93741056,"kb_used":31956,"kb_used_data":776,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93709100,"statfs":{"total":95990841344,"available":95958118400,"internally_reserved":0,"allocated":794624,"data_stored":318436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":30,"seq":128849018919,"num_pgs":58,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93741056,"kb_used":27780,"kb_used_data":760,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93713276,"statfs":{"total":95990841344,"available":95962394624,"internally_reserved":0,"allocated":778240,"data_stored":317262,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757283,"num_pgs":47,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93741056,"kb_used":31920,"kb_used_data":740,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93709136,"statfs":{"total":95990841344,"available":95958155264,"internally_reserved":0,"allocated":757760,"data_stored":317381,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1411,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":921,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":967,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":90112,"data_stored":2338,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":798,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":990,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-10T00:53:28.572 INFO:tasks.cephadm.ceph_manager.ceph:clean! 2024-03-10T00:53:28.572 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-03-10T00:53:28.572 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy 2024-03-10T00:53:28.572 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:831bc188fa69f143bbba69ac76119e784500e8e3 shell --fsid 75c37498-de77-11ee-95c7-87774f69a715 -- ceph health --format=json 2024-03-10T00:53:28.838 INFO:journalctl@ceph.alertmanager.a.smithi106.stdout:Mar 10 00:53:28 smithi106 ceph-75c37498-de77-11ee-95c7-87774f69a715-alertmanager-a[65783]: ts=2024-03-10T00:53:28.593Z caller=cluster.go:698 level=info component=cluster msg="gossip settled; proceeding" elapsed=10.002446764s 2024-03-10T00:53:28.843 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/75c37498-de77-11ee-95c7-87774f69a715/mon.a/config 2024-03-10T00:53:28.975 INFO:journalctl@ceph.iscsi.iscsi.a.smithi170.stdout:Mar 10 00:53:28 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-iscsi-iscsi-a[50134]: debug there is no tcmu-runner data available 2024-03-10T00:53:29.274 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:53:28 smithi170 ceph-mon[27620]: from='client.24662 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-10T00:53:29.274 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:53:28 smithi170 ceph-mon[27620]: pgmap v28: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-10T00:53:29.338 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:53:28 smithi106 ceph-mon[25742]: from='client.24662 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-10T00:53:29.338 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:53:28 smithi106 ceph-mon[25742]: pgmap v28: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-10T00:53:29.338 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:53:28 smithi106 ceph-mon[33460]: from='client.24662 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-10T00:53:29.339 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:53:28 smithi106 ceph-mon[33460]: pgmap v28: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-10T00:53:30.060 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:53:29 smithi106 ceph-mon[25742]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-10T00:53:30.060 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:53:29 smithi106 ceph-mon[33460]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-10T00:53:30.274 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:53:29 smithi170 ceph-mon[27620]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-10T00:53:31.090 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:53:30 smithi106 ceph-mon[25742]: pgmap v29: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-10T00:53:31.090 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:53:30 smithi106 ceph-mon[33460]: pgmap v29: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-10T00:53:31.274 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:53:30 smithi170 ceph-mon[27620]: pgmap v29: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-10T00:53:31.604 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-03-10T00:53:31.604 INFO:teuthology.orchestra.run.smithi106.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-03-10T00:53:31.838 INFO:journalctl@ceph.mgr.y.smithi106.stdout:Mar 10 00:53:31 smithi106 ceph-75c37498-de77-11ee-95c7-87774f69a715-mgr-y[25997]: ::ffff:172.21.15.170 - - [10/Mar/2024:00:53:31] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-10T00:53:32.274 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:53:31 smithi170 ceph-mon[27620]: from='client.? 172.21.15.106:0/961336370' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-03-10T00:53:32.314 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:53:31 smithi106 ceph-mon[33460]: from='client.? 172.21.15.106:0/961336370' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-03-10T00:53:32.314 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:53:31 smithi106 ceph-mon[25742]: from='client.? 172.21.15.106:0/961336370' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-03-10T00:53:32.347 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy done 2024-03-10T00:53:32.348 INFO:tasks.cephadm:Setup complete, yielding 2024-03-10T00:53:32.348 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-03-10T00:53:32.358 INFO:tasks.cephadm:Running commands on role mon.a host ubuntu@smithi106.front.sepia.ceph.com 2024-03-10T00:53:32.359 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:831bc188fa69f143bbba69ac76119e784500e8e3 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 75c37498-de77-11ee-95c7-87774f69a715 -- bash -c 'stat -c '"'"'%u %g'"'"' /var/log/ceph | grep '"'"'167 167'"'"'' 2024-03-10T00:53:32.605 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/75c37498-de77-11ee-95c7-87774f69a715/mon.a/config 2024-03-10T00:53:33.274 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:53:32 smithi170 ceph-mon[27620]: pgmap v30: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-10T00:53:33.338 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:53:32 smithi106 ceph-mon[25742]: pgmap v30: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-10T00:53:33.338 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:53:32 smithi106 ceph-mon[33460]: pgmap v30: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-10T00:53:34.474 INFO:teuthology.orchestra.run.smithi106.stdout:167 167 2024-03-10T00:53:35.088 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:53:34 smithi106 ceph-mon[25742]: pgmap v31: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-10T00:53:35.088 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:53:34 smithi106 ceph-mon[33460]: pgmap v31: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-10T00:53:35.126 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:831bc188fa69f143bbba69ac76119e784500e8e3 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 75c37498-de77-11ee-95c7-87774f69a715 -- bash -c 'ceph orch status' 2024-03-10T00:53:35.274 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:53:34 smithi170 ceph-mon[27620]: pgmap v31: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-10T00:53:35.876 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/75c37498-de77-11ee-95c7-87774f69a715/mon.a/config 2024-03-10T00:53:36.024 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Mar 10 00:53:35 smithi170 systemd[1]: Starting Ceph grafana.a for 75c37498-de77-11ee-95c7-87774f69a715... 2024-03-10T00:53:36.369 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Mar 10 00:53:36 smithi170 podman[55266]: 2024-03-10 00:53:35.954403418 +0000 UTC m=+0.016583915 image pull a0ea5e3c46d9434bd173d585b7c2638c7451eaf2543d1d22f9e78442615aaffb quay.io/ceph/ceph-grafana:9.4.12 2024-03-10T00:53:36.369 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Mar 10 00:53:36 smithi170 podman[55266]: 2024-03-10 00:53:36.061288936 +0000 UTC m=+0.123469420 container create 247603ac339fcf8e08b1a4e78f010d652735b920717e12700166512af49899fe (image=quay.io/ceph/ceph-grafana:9.4.12, name=ceph-75c37498-de77-11ee-95c7-87774f69a715-grafana-a, architecture=x86_64, maintainer=Paul Cuzner , description=Ceph Grafana Container, io.k8s.display-name=Red Hat Universal Base Image 8, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., com.redhat.component=ubi8-container, version=8.5, distribution-scope=public, com.redhat.build-host=cpt-1007.osbs.prod.upshift.rdu2.redhat.com, io.openshift.tags=base rhel8, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, build-date=2022-04-29T11:48:11.055897, io.buildah.version=1.32.0, io.openshift.expose-services=, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-239.1651231664, name=ubi8, release=239.1651231664, vcs-type=git, vendor=Red Hat, Inc., summary=Grafana Container configured for Ceph mgr/dashboard integration) 2024-03-10T00:53:36.369 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Mar 10 00:53:36 smithi170 podman[55266]: 2024-03-10 00:53:36.288543828 +0000 UTC m=+0.350724311 container init 247603ac339fcf8e08b1a4e78f010d652735b920717e12700166512af49899fe (image=quay.io/ceph/ceph-grafana:9.4.12, name=ceph-75c37498-de77-11ee-95c7-87774f69a715-grafana-a, io.openshift.tags=base rhel8, build-date=2022-04-29T11:48:11.055897, release=239.1651231664, io.buildah.version=1.32.0, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, distribution-scope=public, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, maintainer=Paul Cuzner , url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-239.1651231664, vendor=Red Hat, Inc., description=Ceph Grafana Container, io.k8s.display-name=Red Hat Universal Base Image 8, vcs-type=git, io.openshift.expose-services=, com.redhat.component=ubi8-container, com.redhat.build-host=cpt-1007.osbs.prod.upshift.rdu2.redhat.com, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., architecture=x86_64, name=ubi8, summary=Grafana Container configured for Ceph mgr/dashboard integration, version=8.5) 2024-03-10T00:53:36.369 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Mar 10 00:53:36 smithi170 podman[55266]: 2024-03-10 00:53:36.294263335 +0000 UTC m=+0.356443821 container start 247603ac339fcf8e08b1a4e78f010d652735b920717e12700166512af49899fe (image=quay.io/ceph/ceph-grafana:9.4.12, name=ceph-75c37498-de77-11ee-95c7-87774f69a715-grafana-a, architecture=x86_64, vcs-type=git, io.openshift.expose-services=, com.redhat.build-host=cpt-1007.osbs.prod.upshift.rdu2.redhat.com, summary=Grafana Container configured for Ceph mgr/dashboard integration, version=8.5, io.buildah.version=1.32.0, distribution-scope=public, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-239.1651231664, release=239.1651231664, name=ubi8, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, build-date=2022-04-29T11:48:11.055897, vendor=Red Hat, Inc., description=Ceph Grafana Container, io.k8s.display-name=Red Hat Universal Base Image 8, com.redhat.component=ubi8-container, maintainer=Paul Cuzner , io.openshift.tags=base rhel8, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly.) 2024-03-10T00:53:36.369 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Mar 10 00:53:36 smithi170 bash[55266]: 247603ac339fcf8e08b1a4e78f010d652735b920717e12700166512af49899fe 2024-03-10T00:53:36.620 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Mar 10 00:53:36 smithi170 systemd[1]: Started Ceph grafana.a for 75c37498-de77-11ee-95c7-87774f69a715. 2024-03-10T00:53:36.620 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Mar 10 00:53:36 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-grafana-a[55281]: logger=settings t=2024-03-10T00:53:36.460377697Z level=info msg="Starting Grafana" version=9.4.12 commit=37ee25ba46 branch=HEAD compiled=2023-05-22T15:58:05Z 2024-03-10T00:53:36.620 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Mar 10 00:53:36 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-grafana-a[55281]: logger=settings t=2024-03-10T00:53:36.460622815Z level=warn msg="\"sentry\" frontend logging provider is deprecated and will be removed in the next major version. Use \"grafana\" provider instead." 2024-03-10T00:53:36.620 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Mar 10 00:53:36 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-grafana-a[55281]: logger=settings t=2024-03-10T00:53:36.460649286Z level=info msg="Config loaded from" file=/usr/share/grafana/conf/defaults.ini 2024-03-10T00:53:36.620 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Mar 10 00:53:36 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-grafana-a[55281]: logger=settings t=2024-03-10T00:53:36.460659563Z level=info msg="Config loaded from" file=/etc/grafana/grafana.ini 2024-03-10T00:53:36.621 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Mar 10 00:53:36 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-grafana-a[55281]: logger=settings t=2024-03-10T00:53:36.46066959Z level=info msg="Config overridden from Environment variable" var="GF_PATHS_DATA=/var/lib/grafana" 2024-03-10T00:53:36.621 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Mar 10 00:53:36 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-grafana-a[55281]: logger=settings t=2024-03-10T00:53:36.460679491Z level=info msg="Config overridden from Environment variable" var="GF_PATHS_LOGS=/var/log/grafana" 2024-03-10T00:53:36.621 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Mar 10 00:53:36 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-grafana-a[55281]: logger=settings t=2024-03-10T00:53:36.460687641Z level=info msg="Config overridden from Environment variable" var="GF_PATHS_PLUGINS=/var/lib/grafana/plugins" 2024-03-10T00:53:36.621 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Mar 10 00:53:36 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-grafana-a[55281]: logger=settings t=2024-03-10T00:53:36.460696744Z level=info msg="Config overridden from Environment variable" var="GF_PATHS_PROVISIONING=/etc/grafana/provisioning" 2024-03-10T00:53:36.621 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Mar 10 00:53:36 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-grafana-a[55281]: logger=settings t=2024-03-10T00:53:36.460706961Z level=info msg="Path Home" path=/usr/share/grafana 2024-03-10T00:53:36.621 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Mar 10 00:53:36 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-grafana-a[55281]: logger=settings t=2024-03-10T00:53:36.460716199Z level=info msg="Path Data" path=/var/lib/grafana 2024-03-10T00:53:36.621 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Mar 10 00:53:36 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-grafana-a[55281]: logger=settings t=2024-03-10T00:53:36.460725328Z level=info msg="Path Logs" path=/var/log/grafana 2024-03-10T00:53:36.621 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Mar 10 00:53:36 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-grafana-a[55281]: logger=settings t=2024-03-10T00:53:36.460733273Z level=info msg="Path Plugins" path=/var/lib/grafana/plugins 2024-03-10T00:53:36.621 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Mar 10 00:53:36 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-grafana-a[55281]: logger=settings t=2024-03-10T00:53:36.460741403Z level=info msg="Path Provisioning" path=/etc/grafana/provisioning 2024-03-10T00:53:36.621 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Mar 10 00:53:36 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-grafana-a[55281]: logger=settings t=2024-03-10T00:53:36.460749754Z level=info msg="App mode production" 2024-03-10T00:53:36.621 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Mar 10 00:53:36 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-grafana-a[55281]: logger=sqlstore t=2024-03-10T00:53:36.46081119Z level=info msg="Connecting to DB" dbtype=sqlite3 2024-03-10T00:53:36.621 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Mar 10 00:53:36 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-grafana-a[55281]: logger=sqlstore t=2024-03-10T00:53:36.460835416Z level=warn msg="SQLite database file has broader permissions than it should" path=/var/lib/grafana/grafana.db mode=-rw-r--r-- expected=-rw-r----- 2024-03-10T00:53:36.622 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Mar 10 00:53:36 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-grafana-a[55281]: logger=migrator t=2024-03-10T00:53:36.461791851Z level=info msg="Starting DB migrations" 2024-03-10T00:53:36.622 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Mar 10 00:53:36 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-grafana-a[55281]: logger=migrator t=2024-03-10T00:53:36.461881633Z level=info msg="Executing migration" id="create migration_log table" 2024-03-10T00:53:36.623 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Mar 10 00:53:36 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-grafana-a[55281]: logger=migrator t=2024-03-10T00:53:36.462425706Z level=info msg="Executing migration" id="create user table" 2024-03-10T00:53:36.623 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Mar 10 00:53:36 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-grafana-a[55281]: logger=migrator t=2024-03-10T00:53:36.462875783Z level=info msg="Executing migration" id="add unique index user.login" 2024-03-10T00:53:36.623 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Mar 10 00:53:36 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-grafana-a[55281]: logger=migrator t=2024-03-10T00:53:36.463536502Z level=info msg="Executing migration" id="add unique index user.email" 2024-03-10T00:53:36.623 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Mar 10 00:53:36 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-grafana-a[55281]: logger=migrator t=2024-03-10T00:53:36.46396746Z level=info msg="Executing migration" id="drop index UQE_user_login - v1" 2024-03-10T00:53:36.623 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Mar 10 00:53:36 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-grafana-a[55281]: logger=migrator t=2024-03-10T00:53:36.464510068Z level=info msg="Executing migration" id="drop index UQE_user_email - v1" 2024-03-10T00:53:36.623 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Mar 10 00:53:36 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-grafana-a[55281]: logger=migrator t=2024-03-10T00:53:36.465006242Z level=info msg="Executing migration" id="Rename table user to user_v1 - v1" 2024-03-10T00:53:36.625 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Mar 10 00:53:36 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-grafana-a[55281]: logger=migrator t=2024-03-10T00:53:36.466016667Z level=info msg="Executing migration" id="create user table v2" 2024-03-10T00:53:36.625 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Mar 10 00:53:36 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-grafana-a[55281]: logger=migrator t=2024-03-10T00:53:36.466572662Z level=info msg="Executing migration" id="create index UQE_user_login - v2" 2024-03-10T00:53:36.625 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Mar 10 00:53:36 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-grafana-a[55281]: logger=migrator t=2024-03-10T00:53:36.467066691Z level=info msg="Executing migration" id="create index UQE_user_email - v2" 2024-03-10T00:53:36.625 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Mar 10 00:53:36 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-grafana-a[55281]: logger=migrator t=2024-03-10T00:53:36.467623602Z level=info msg="Executing migration" id="copy data_source v1 to v2" 2024-03-10T00:53:36.625 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Mar 10 00:53:36 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-grafana-a[55281]: logger=migrator t=2024-03-10T00:53:36.467946583Z level=info msg="Executing migration" id="Drop old table user_v1" 2024-03-10T00:53:36.625 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Mar 10 00:53:36 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-grafana-a[55281]: logger=migrator t=2024-03-10T00:53:36.46831424Z level=info msg="Executing migration" id="Add column help_flags1 to user table" 2024-03-10T00:53:36.626 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Mar 10 00:53:36 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-grafana-a[55281]: logger=migrator t=2024-03-10T00:53:36.468725212Z level=info msg="Executing migration" id="Update user table charset" 2024-03-10T00:53:36.626 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Mar 10 00:53:36 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-grafana-a[55281]: logger=migrator t=2024-03-10T00:53:36.468964634Z level=info msg="Executing migration" id="Add last_seen_at column to user" 2024-03-10T00:53:36.626 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Mar 10 00:53:36 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-grafana-a[55281]: logger=migrator t=2024-03-10T00:53:36.469359228Z level=info msg="Executing migration" id="Add missing user data" 2024-03-10T00:53:36.626 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Mar 10 00:53:36 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-grafana-a[55281]: logger=migrator t=2024-03-10T00:53:36.469691518Z level=info msg="Executing migration" id="Add is_disabled column to user" 2024-03-10T00:53:36.626 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Mar 10 00:53:36 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-grafana-a[55281]: logger=migrator t=2024-03-10T00:53:36.470063229Z level=info msg="Executing migration" id="Add index user.login/user.email" 2024-03-10T00:53:36.626 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Mar 10 00:53:36 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-grafana-a[55281]: logger=migrator t=2024-03-10T00:53:36.470479031Z level=info msg="Executing migration" id="Add is_service_account column to user" 2024-03-10T00:53:36.626 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Mar 10 00:53:36 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-grafana-a[55281]: logger=migrator t=2024-03-10T00:53:36.470869173Z level=info msg="Executing migration" id="Update is_service_account column to nullable" 2024-03-10T00:53:36.626 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Mar 10 00:53:36 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-grafana-a[55281]: logger=migrator t=2024-03-10T00:53:36.472015078Z level=info msg="Executing migration" id="create temp user table v1-7" 2024-03-10T00:53:36.626 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Mar 10 00:53:36 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-grafana-a[55281]: logger=migrator t=2024-03-10T00:53:36.472478121Z level=info msg="Executing migration" id="create index IDX_temp_user_email - v1-7" 2024-03-10T00:53:36.626 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Mar 10 00:53:36 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-grafana-a[55281]: logger=migrator t=2024-03-10T00:53:36.472939265Z level=info msg="Executing migration" id="create index IDX_temp_user_org_id - v1-7" 2024-03-10T00:53:36.626 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Mar 10 00:53:36 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-grafana-a[55281]: logger=migrator t=2024-03-10T00:53:36.473343539Z level=info msg="Executing migration" id="create index IDX_temp_user_code - v1-7" 2024-03-10T00:53:36.627 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Mar 10 00:53:36 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-grafana-a[55281]: logger=migrator t=2024-03-10T00:53:36.473735409Z level=info msg="Executing migration" id="create index IDX_temp_user_status - v1-7" 2024-03-10T00:53:36.627 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Mar 10 00:53:36 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-grafana-a[55281]: logger=migrator t=2024-03-10T00:53:36.474125984Z level=info msg="Executing migration" id="Update temp_user table charset" 2024-03-10T00:53:36.627 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Mar 10 00:53:36 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-grafana-a[55281]: logger=migrator t=2024-03-10T00:53:36.474423015Z level=info msg="Executing migration" id="drop index IDX_temp_user_email - v1" 2024-03-10T00:53:36.627 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Mar 10 00:53:36 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-grafana-a[55281]: logger=migrator t=2024-03-10T00:53:36.47477142Z level=info msg="Executing migration" id="drop index IDX_temp_user_org_id - v1" 2024-03-10T00:53:36.627 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Mar 10 00:53:36 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-grafana-a[55281]: logger=migrator t=2024-03-10T00:53:36.475125959Z level=info msg="Executing migration" id="drop index IDX_temp_user_code - v1" 2024-03-10T00:53:36.627 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Mar 10 00:53:36 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-grafana-a[55281]: logger=migrator t=2024-03-10T00:53:36.47565712Z level=info msg="Executing migration" id="drop index IDX_temp_user_status - v1" 2024-03-10T00:53:36.627 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Mar 10 00:53:36 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-grafana-a[55281]: logger=migrator t=2024-03-10T00:53:36.476020523Z level=info msg="Executing migration" id="Rename table temp_user to temp_user_tmp_qwerty - v1" 2024-03-10T00:53:36.627 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Mar 10 00:53:36 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-grafana-a[55281]: logger=migrator t=2024-03-10T00:53:36.476892866Z level=info msg="Executing migration" id="create temp_user v2" 2024-03-10T00:53:36.627 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Mar 10 00:53:36 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-grafana-a[55281]: logger=migrator t=2024-03-10T00:53:36.477352247Z level=info msg="Executing migration" id="create index IDX_temp_user_email - v2" 2024-03-10T00:53:36.627 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Mar 10 00:53:36 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-grafana-a[55281]: logger=migrator t=2024-03-10T00:53:36.477734993Z level=info msg="Executing migration" id="create index IDX_temp_user_org_id - v2" 2024-03-10T00:53:36.628 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Mar 10 00:53:36 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-grafana-a[55281]: logger=migrator t=2024-03-10T00:53:36.478069744Z level=info msg="Executing migration" id="create index IDX_temp_user_code - v2" 2024-03-10T00:53:36.628 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Mar 10 00:53:36 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-grafana-a[55281]: logger=migrator t=2024-03-10T00:53:36.47844657Z level=info msg="Executing migration" id="create index IDX_temp_user_status - v2" 2024-03-10T00:53:36.628 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Mar 10 00:53:36 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-grafana-a[55281]: logger=migrator t=2024-03-10T00:53:36.478852661Z level=info msg="Executing migration" id="copy temp_user v1 to v2" 2024-03-10T00:53:36.628 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Mar 10 00:53:36 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-grafana-a[55281]: logger=migrator t=2024-03-10T00:53:36.479136426Z level=info msg="Executing migration" id="drop temp_user_tmp_qwerty" 2024-03-10T00:53:36.628 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Mar 10 00:53:36 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-grafana-a[55281]: logger=migrator t=2024-03-10T00:53:36.479452815Z level=info msg="Executing migration" id="Set created for temp users that will otherwise prematurely expire" 2024-03-10T00:53:36.628 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Mar 10 00:53:36 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-grafana-a[55281]: logger=migrator t=2024-03-10T00:53:36.479737712Z level=info msg="Executing migration" id="create star table" 2024-03-10T00:53:36.628 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Mar 10 00:53:36 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-grafana-a[55281]: logger=migrator t=2024-03-10T00:53:36.480043827Z level=info msg="Executing migration" id="add unique index star.user_id_dashboard_id" 2024-03-10T00:53:36.628 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Mar 10 00:53:36 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-grafana-a[55281]: logger=migrator t=2024-03-10T00:53:36.480496125Z level=info msg="Executing migration" id="create org table v1" 2024-03-10T00:53:36.628 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Mar 10 00:53:36 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-grafana-a[55281]: logger=migrator t=2024-03-10T00:53:36.48092439Z level=info msg="Executing migration" id="create index UQE_org_name - v1" 2024-03-10T00:53:36.629 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Mar 10 00:53:36 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-grafana-a[55281]: logger=migrator t=2024-03-10T00:53:36.481324019Z level=info msg="Executing migration" id="create org_user table v1" 2024-03-10T00:53:36.629 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Mar 10 00:53:36 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-grafana-a[55281]: logger=migrator t=2024-03-10T00:53:36.481667261Z level=info msg="Executing migration" id="create index IDX_org_user_org_id - v1" 2024-03-10T00:53:36.629 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Mar 10 00:53:36 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-grafana-a[55281]: logger=migrator t=2024-03-10T00:53:36.482018354Z level=info msg="Executing migration" id="create index UQE_org_user_org_id_user_id - v1" 2024-03-10T00:53:36.629 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Mar 10 00:53:36 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-grafana-a[55281]: logger=migrator t=2024-03-10T00:53:36.482447137Z level=info msg="Executing migration" id="create index IDX_org_user_user_id - v1" 2024-03-10T00:53:36.629 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Mar 10 00:53:36 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-grafana-a[55281]: logger=migrator t=2024-03-10T00:53:36.482876805Z level=info msg="Executing migration" id="Update org table charset" 2024-03-10T00:53:36.629 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Mar 10 00:53:36 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-grafana-a[55281]: logger=migrator t=2024-03-10T00:53:36.483102667Z level=info msg="Executing migration" id="Update org_user table charset" 2024-03-10T00:53:36.629 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Mar 10 00:53:36 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-grafana-a[55281]: logger=migrator t=2024-03-10T00:53:36.48342426Z level=info msg="Executing migration" id="Migrate all Read Only Viewers to Viewers" 2024-03-10T00:53:36.629 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Mar 10 00:53:36 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-grafana-a[55281]: logger=migrator t=2024-03-10T00:53:36.483677778Z level=info msg="Executing migration" id="create dashboard table" 2024-03-10T00:53:36.629 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Mar 10 00:53:36 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-grafana-a[55281]: logger=migrator t=2024-03-10T00:53:36.48401873Z level=info msg="Executing migration" id="add index dashboard.account_id" 2024-03-10T00:53:36.629 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Mar 10 00:53:36 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-grafana-a[55281]: logger=migrator t=2024-03-10T00:53:36.48442082Z level=info msg="Executing migration" id="add unique index dashboard_account_id_slug" 2024-03-10T00:53:36.629 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Mar 10 00:53:36 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-grafana-a[55281]: logger=migrator t=2024-03-10T00:53:36.484839019Z level=info msg="Executing migration" id="create dashboard_tag table" 2024-03-10T00:53:36.630 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Mar 10 00:53:36 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-grafana-a[55281]: logger=migrator t=2024-03-10T00:53:36.485409718Z level=info msg="Executing migration" id="add unique index dashboard_tag.dasboard_id_term" 2024-03-10T00:53:36.630 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Mar 10 00:53:36 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-grafana-a[55281]: logger=migrator t=2024-03-10T00:53:36.485942289Z level=info msg="Executing migration" id="drop index UQE_dashboard_tag_dashboard_id_term - v1" 2024-03-10T00:53:36.630 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Mar 10 00:53:36 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-grafana-a[55281]: logger=migrator t=2024-03-10T00:53:36.486457208Z level=info msg="Executing migration" id="Rename table dashboard to dashboard_v1 - v1" 2024-03-10T00:53:36.630 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Mar 10 00:53:36 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-grafana-a[55281]: logger=migrator t=2024-03-10T00:53:36.487799309Z level=info msg="Executing migration" id="create dashboard v2" 2024-03-10T00:53:36.630 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Mar 10 00:53:36 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-grafana-a[55281]: logger=migrator t=2024-03-10T00:53:36.488318418Z level=info msg="Executing migration" id="create index IDX_dashboard_org_id - v2" 2024-03-10T00:53:36.630 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Mar 10 00:53:36 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-grafana-a[55281]: logger=migrator t=2024-03-10T00:53:36.488851543Z level=info msg="Executing migration" id="create index UQE_dashboard_org_id_slug - v2" 2024-03-10T00:53:36.631 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Mar 10 00:53:36 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-grafana-a[55281]: logger=migrator t=2024-03-10T00:53:36.489396861Z level=info msg="Executing migration" id="copy dashboard v1 to v2" 2024-03-10T00:53:36.631 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Mar 10 00:53:36 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-grafana-a[55281]: logger=migrator t=2024-03-10T00:53:36.489809978Z level=info msg="Executing migration" id="drop table dashboard_v1" 2024-03-10T00:53:36.631 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Mar 10 00:53:36 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-grafana-a[55281]: logger=migrator t=2024-03-10T00:53:36.490331034Z level=info msg="Executing migration" id="alter dashboard.data to mediumtext v1" 2024-03-10T00:53:36.631 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Mar 10 00:53:36 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-grafana-a[55281]: logger=migrator t=2024-03-10T00:53:36.490767481Z level=info msg="Executing migration" id="Add column updated_by in dashboard - v2" 2024-03-10T00:53:36.631 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Mar 10 00:53:36 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-grafana-a[55281]: logger=migrator t=2024-03-10T00:53:36.49147261Z level=info msg="Executing migration" id="Add column created_by in dashboard - v2" 2024-03-10T00:53:36.631 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Mar 10 00:53:36 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-grafana-a[55281]: logger=migrator t=2024-03-10T00:53:36.492148884Z level=info msg="Executing migration" id="Add column gnetId in dashboard" 2024-03-10T00:53:36.631 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Mar 10 00:53:36 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-grafana-a[55281]: logger=migrator t=2024-03-10T00:53:36.4928072Z level=info msg="Executing migration" id="Add index for gnetId in dashboard" 2024-03-10T00:53:36.631 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Mar 10 00:53:36 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-grafana-a[55281]: logger=migrator t=2024-03-10T00:53:36.493352116Z level=info msg="Executing migration" id="Add column plugin_id in dashboard" 2024-03-10T00:53:36.631 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Mar 10 00:53:36 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-grafana-a[55281]: logger=migrator t=2024-03-10T00:53:36.494006926Z level=info msg="Executing migration" id="Add index for plugin_id in dashboard" 2024-03-10T00:53:36.632 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Mar 10 00:53:36 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-grafana-a[55281]: logger=migrator t=2024-03-10T00:53:36.494589361Z level=info msg="Executing migration" id="Add index for dashboard_id in dashboard_tag" 2024-03-10T00:53:36.632 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Mar 10 00:53:36 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-grafana-a[55281]: logger=migrator t=2024-03-10T00:53:36.495121939Z level=info msg="Executing migration" id="Update dashboard table charset" 2024-03-10T00:53:36.632 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Mar 10 00:53:36 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-grafana-a[55281]: logger=migrator t=2024-03-10T00:53:36.495477364Z level=info msg="Executing migration" id="Update dashboard_tag table charset" 2024-03-10T00:53:36.632 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Mar 10 00:53:36 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-grafana-a[55281]: logger=migrator t=2024-03-10T00:53:36.495794888Z level=info msg="Executing migration" id="Add column folder_id in dashboard" 2024-03-10T00:53:36.632 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Mar 10 00:53:36 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-grafana-a[55281]: logger=migrator t=2024-03-10T00:53:36.496489795Z level=info msg="Executing migration" id="Add column isFolder in dashboard" 2024-03-10T00:53:36.632 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Mar 10 00:53:36 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-grafana-a[55281]: logger=migrator t=2024-03-10T00:53:36.49694814Z level=info msg="Executing migration" id="Add column has_acl in dashboard" 2024-03-10T00:53:36.632 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Mar 10 00:53:36 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-grafana-a[55281]: logger=migrator t=2024-03-10T00:53:36.497408931Z level=info msg="Executing migration" id="Add column uid in dashboard" 2024-03-10T00:53:36.632 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Mar 10 00:53:36 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-grafana-a[55281]: logger=migrator t=2024-03-10T00:53:36.497869762Z level=info msg="Executing migration" id="Update uid column values in dashboard" 2024-03-10T00:53:36.632 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Mar 10 00:53:36 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-grafana-a[55281]: logger=migrator t=2024-03-10T00:53:36.498127316Z level=info msg="Executing migration" id="Add unique index dashboard_org_id_uid" 2024-03-10T00:53:36.632 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Mar 10 00:53:36 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-grafana-a[55281]: logger=migrator t=2024-03-10T00:53:36.498499521Z level=info msg="Executing migration" id="Remove unique index org_id_slug" 2024-03-10T00:53:36.635 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Mar 10 00:53:36 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-grafana-a[55281]: logger=migrator t=2024-03-10T00:53:36.49882488Z level=info msg="Executing migration" id="Update dashboard title length" 2024-03-10T00:53:36.635 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Mar 10 00:53:36 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-grafana-a[55281]: logger=migrator t=2024-03-10T00:53:36.499044659Z level=info msg="Executing migration" id="Add unique index for dashboard_org_id_title_folder_id" 2024-03-10T00:53:36.635 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Mar 10 00:53:36 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-grafana-a[55281]: logger=migrator t=2024-03-10T00:53:36.499433471Z level=info msg="Executing migration" id="create dashboard_provisioning" 2024-03-10T00:53:36.635 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Mar 10 00:53:36 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-grafana-a[55281]: logger=migrator t=2024-03-10T00:53:36.499777003Z level=info msg="Executing migration" id="Rename table dashboard_provisioning to dashboard_provisioning_tmp_qwerty - v1" 2024-03-10T00:53:36.635 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Mar 10 00:53:36 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-grafana-a[55281]: logger=migrator t=2024-03-10T00:53:36.500758519Z level=info msg="Executing migration" id="create dashboard_provisioning v2" 2024-03-10T00:53:36.635 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Mar 10 00:53:36 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-grafana-a[55281]: logger=migrator t=2024-03-10T00:53:36.501106069Z level=info msg="Executing migration" id="create index IDX_dashboard_provisioning_dashboard_id - v2" 2024-03-10T00:53:36.635 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Mar 10 00:53:36 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-grafana-a[55281]: logger=migrator t=2024-03-10T00:53:36.501494111Z level=info msg="Executing migration" id="create index IDX_dashboard_provisioning_dashboard_id_name - v2" 2024-03-10T00:53:36.635 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Mar 10 00:53:36 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-grafana-a[55281]: logger=migrator t=2024-03-10T00:53:36.501879747Z level=info msg="Executing migration" id="copy dashboard_provisioning v1 to v2" 2024-03-10T00:53:36.635 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Mar 10 00:53:36 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-grafana-a[55281]: logger=migrator t=2024-03-10T00:53:36.502160026Z level=info msg="Executing migration" id="drop dashboard_provisioning_tmp_qwerty" 2024-03-10T00:53:36.635 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Mar 10 00:53:36 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-grafana-a[55281]: logger=migrator t=2024-03-10T00:53:36.502509933Z level=info msg="Executing migration" id="Add check_sum column" 2024-03-10T00:53:36.635 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Mar 10 00:53:36 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-grafana-a[55281]: logger=migrator t=2024-03-10T00:53:36.5041479Z level=info msg="Executing migration" id="Add index for dashboard_title" 2024-03-10T00:53:36.636 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Mar 10 00:53:36 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-grafana-a[55281]: logger=migrator t=2024-03-10T00:53:36.504587266Z level=info msg="Executing migration" id="delete tags for deleted dashboards" 2024-03-10T00:53:36.636 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Mar 10 00:53:36 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-grafana-a[55281]: logger=migrator t=2024-03-10T00:53:36.504842599Z level=info msg="Executing migration" id="delete stars for deleted dashboards" 2024-03-10T00:53:36.636 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Mar 10 00:53:36 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-grafana-a[55281]: logger=migrator t=2024-03-10T00:53:36.50509102Z level=info msg="Executing migration" id="Add index for dashboard_is_folder" 2024-03-10T00:53:36.636 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Mar 10 00:53:36 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-grafana-a[55281]: logger=migrator t=2024-03-10T00:53:36.505532969Z level=info msg="Executing migration" id="Add isPublic for dashboard" 2024-03-10T00:53:36.636 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Mar 10 00:53:36 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-grafana-a[55281]: logger=migrator t=2024-03-10T00:53:36.506011104Z level=info msg="Executing migration" id="create data_source table" 2024-03-10T00:53:36.636 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Mar 10 00:53:36 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-grafana-a[55281]: logger=migrator t=2024-03-10T00:53:36.506440529Z level=info msg="Executing migration" id="add index data_source.account_id" 2024-03-10T00:53:36.636 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Mar 10 00:53:36 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-grafana-a[55281]: logger=migrator t=2024-03-10T00:53:36.50686876Z level=info msg="Executing migration" id="add unique index data_source.account_id_name" 2024-03-10T00:53:36.636 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Mar 10 00:53:36 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-grafana-a[55281]: logger=migrator t=2024-03-10T00:53:36.507291777Z level=info msg="Executing migration" id="drop index IDX_data_source_account_id - v1" 2024-03-10T00:53:36.636 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Mar 10 00:53:36 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-grafana-a[55281]: logger=migrator t=2024-03-10T00:53:36.507670143Z level=info msg="Executing migration" id="drop index UQE_data_source_account_id_name - v1" 2024-03-10T00:53:36.636 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Mar 10 00:53:36 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-grafana-a[55281]: logger=migrator t=2024-03-10T00:53:36.508067001Z level=info msg="Executing migration" id="Rename table data_source to data_source_v1 - v1" 2024-03-10T00:53:36.636 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Mar 10 00:53:36 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-grafana-a[55281]: logger=migrator t=2024-03-10T00:53:36.509010304Z level=info msg="Executing migration" id="create data_source table v2" 2024-03-10T00:53:36.636 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Mar 10 00:53:36 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-grafana-a[55281]: logger=migrator t=2024-03-10T00:53:36.509398072Z level=info msg="Executing migration" id="create index IDX_data_source_org_id - v2" 2024-03-10T00:53:36.636 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Mar 10 00:53:36 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-grafana-a[55281]: logger=migrator t=2024-03-10T00:53:36.509792098Z level=info msg="Executing migration" id="create index UQE_data_source_org_id_name - v2" 2024-03-10T00:53:36.636 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Mar 10 00:53:36 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-grafana-a[55281]: logger=migrator t=2024-03-10T00:53:36.510174847Z level=info msg="Executing migration" id="Drop old table data_source_v1 #2" 2024-03-10T00:53:36.637 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Mar 10 00:53:36 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-grafana-a[55281]: logger=migrator t=2024-03-10T00:53:36.510510906Z level=info msg="Executing migration" id="Add column with_credentials" 2024-03-10T00:53:36.637 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Mar 10 00:53:36 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-grafana-a[55281]: logger=migrator t=2024-03-10T00:53:36.510989488Z level=info msg="Executing migration" id="Add secure json data column" 2024-03-10T00:53:36.637 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Mar 10 00:53:36 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-grafana-a[55281]: logger=migrator t=2024-03-10T00:53:36.511531676Z level=info msg="Executing migration" id="Update data_source table charset" 2024-03-10T00:53:36.637 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Mar 10 00:53:36 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-grafana-a[55281]: logger=migrator t=2024-03-10T00:53:36.511792761Z level=info msg="Executing migration" id="Update initial version to 1" 2024-03-10T00:53:36.637 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Mar 10 00:53:36 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-grafana-a[55281]: logger=migrator t=2024-03-10T00:53:36.512038748Z level=info msg="Executing migration" id="Add read_only data column" 2024-03-10T00:53:36.637 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Mar 10 00:53:36 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-grafana-a[55281]: logger=migrator t=2024-03-10T00:53:36.512530497Z level=info msg="Executing migration" id="Migrate logging ds to loki ds" 2024-03-10T00:53:36.637 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Mar 10 00:53:36 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-grafana-a[55281]: logger=migrator t=2024-03-10T00:53:36.512842132Z level=info msg="Executing migration" id="Update json_data with nulls" 2024-03-10T00:53:36.637 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Mar 10 00:53:36 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-grafana-a[55281]: logger=migrator t=2024-03-10T00:53:36.513097254Z level=info msg="Executing migration" id="Add uid column" 2024-03-10T00:53:36.637 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Mar 10 00:53:36 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-grafana-a[55281]: logger=migrator t=2024-03-10T00:53:36.513642599Z level=info msg="Executing migration" id="Update uid value" 2024-03-10T00:53:36.637 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Mar 10 00:53:36 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-grafana-a[55281]: logger=migrator t=2024-03-10T00:53:36.513901167Z level=info msg="Executing migration" id="Add unique index datasource_org_id_uid" 2024-03-10T00:53:36.637 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Mar 10 00:53:36 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-grafana-a[55281]: logger=migrator t=2024-03-10T00:53:36.514308311Z level=info msg="Executing migration" id="add unique index datasource_org_id_is_default" 2024-03-10T00:53:36.637 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Mar 10 00:53:36 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-grafana-a[55281]: logger=migrator t=2024-03-10T00:53:36.514700172Z level=info msg="Executing migration" id="create api_key table" 2024-03-10T00:53:36.637 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Mar 10 00:53:36 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-grafana-a[55281]: logger=migrator t=2024-03-10T00:53:36.515134098Z level=info msg="Executing migration" id="add index api_key.account_id" 2024-03-10T00:53:36.637 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Mar 10 00:53:36 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-grafana-a[55281]: logger=migrator t=2024-03-10T00:53:36.515566169Z level=info msg="Executing migration" id="add index api_key.key" 2024-03-10T00:53:36.637 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Mar 10 00:53:36 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-grafana-a[55281]: logger=migrator t=2024-03-10T00:53:36.51596285Z level=info msg="Executing migration" id="add index api_key.account_id_name" 2024-03-10T00:53:36.638 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Mar 10 00:53:36 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-grafana-a[55281]: logger=migrator t=2024-03-10T00:53:36.516418234Z level=info msg="Executing migration" id="drop index IDX_api_key_account_id - v1" 2024-03-10T00:53:36.638 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Mar 10 00:53:36 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-grafana-a[55281]: logger=migrator t=2024-03-10T00:53:36.516795568Z level=info msg="Executing migration" id="drop index UQE_api_key_key - v1" 2024-03-10T00:53:36.638 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Mar 10 00:53:36 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-grafana-a[55281]: logger=migrator t=2024-03-10T00:53:36.517159179Z level=info msg="Executing migration" id="drop index UQE_api_key_account_id_name - v1" 2024-03-10T00:53:36.638 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Mar 10 00:53:36 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-grafana-a[55281]: logger=migrator t=2024-03-10T00:53:36.517567755Z level=info msg="Executing migration" id="Rename table api_key to api_key_v1 - v1" 2024-03-10T00:53:36.638 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Mar 10 00:53:36 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-grafana-a[55281]: logger=migrator t=2024-03-10T00:53:36.518652214Z level=info msg="Executing migration" id="create api_key table v2" 2024-03-10T00:53:36.638 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Mar 10 00:53:36 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-grafana-a[55281]: logger=migrator t=2024-03-10T00:53:36.519021656Z level=info msg="Executing migration" id="create index IDX_api_key_org_id - v2" 2024-03-10T00:53:36.638 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Mar 10 00:53:36 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-grafana-a[55281]: logger=migrator t=2024-03-10T00:53:36.519422342Z level=info msg="Executing migration" id="create index UQE_api_key_key - v2" 2024-03-10T00:53:36.638 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Mar 10 00:53:36 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-grafana-a[55281]: logger=migrator t=2024-03-10T00:53:36.519809399Z level=info msg="Executing migration" id="create index UQE_api_key_org_id_name - v2" 2024-03-10T00:53:36.638 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Mar 10 00:53:36 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-grafana-a[55281]: logger=migrator t=2024-03-10T00:53:36.520209365Z level=info msg="Executing migration" id="copy api_key v1 to v2" 2024-03-10T00:53:36.638 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Mar 10 00:53:36 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-grafana-a[55281]: logger=migrator t=2024-03-10T00:53:36.520511296Z level=info msg="Executing migration" id="Drop old table api_key_v1" 2024-03-10T00:53:36.638 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Mar 10 00:53:36 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-grafana-a[55281]: logger=migrator t=2024-03-10T00:53:36.520807609Z level=info msg="Executing migration" id="Update api_key table charset" 2024-03-10T00:53:36.638 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Mar 10 00:53:36 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-grafana-a[55281]: logger=migrator t=2024-03-10T00:53:36.521039374Z level=info msg="Executing migration" id="Add expires to api_key table" 2024-03-10T00:53:36.638 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Mar 10 00:53:36 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-grafana-a[55281]: logger=migrator t=2024-03-10T00:53:36.521572135Z level=info msg="Executing migration" id="Add service account foreign key" 2024-03-10T00:53:36.638 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Mar 10 00:53:36 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-grafana-a[55281]: logger=migrator t=2024-03-10T00:53:36.52206802Z level=info msg="Executing migration" id="set service account foreign key to nil if 0" 2024-03-10T00:53:36.639 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Mar 10 00:53:36 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-grafana-a[55281]: logger=migrator t=2024-03-10T00:53:36.522375922Z level=info msg="Executing migration" id="Add last_used_at to api_key table" 2024-03-10T00:53:36.639 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Mar 10 00:53:36 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-grafana-a[55281]: logger=migrator t=2024-03-10T00:53:36.523397536Z level=info msg="Executing migration" id="Add is_revoked column to api_key table" 2024-03-10T00:53:36.639 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Mar 10 00:53:36 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-grafana-a[55281]: logger=migrator t=2024-03-10T00:53:36.523998546Z level=info msg="Executing migration" id="create dashboard_snapshot table v4" 2024-03-10T00:53:36.639 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Mar 10 00:53:36 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-grafana-a[55281]: logger=migrator t=2024-03-10T00:53:36.524368149Z level=info msg="Executing migration" id="drop table dashboard_snapshot_v4 #1" 2024-03-10T00:53:36.639 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Mar 10 00:53:36 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-grafana-a[55281]: logger=migrator t=2024-03-10T00:53:36.524745256Z level=info msg="Executing migration" id="create dashboard_snapshot table v5 #2" 2024-03-10T00:53:36.639 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Mar 10 00:53:36 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-grafana-a[55281]: logger=migrator t=2024-03-10T00:53:36.525095935Z level=info msg="Executing migration" id="create index UQE_dashboard_snapshot_key - v5" 2024-03-10T00:53:36.639 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Mar 10 00:53:36 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-grafana-a[55281]: logger=migrator t=2024-03-10T00:53:36.525511524Z level=info msg="Executing migration" id="create index UQE_dashboard_snapshot_delete_key - v5" 2024-03-10T00:53:36.639 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Mar 10 00:53:36 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-grafana-a[55281]: logger=migrator t=2024-03-10T00:53:36.525899271Z level=info msg="Executing migration" id="create index IDX_dashboard_snapshot_user_id - v5" 2024-03-10T00:53:36.639 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Mar 10 00:53:36 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-grafana-a[55281]: logger=migrator t=2024-03-10T00:53:36.526286618Z level=info msg="Executing migration" id="alter dashboard_snapshot to mediumtext v2" 2024-03-10T00:53:36.639 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Mar 10 00:53:36 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-grafana-a[55281]: logger=migrator t=2024-03-10T00:53:36.526560832Z level=info msg="Executing migration" id="Update dashboard_snapshot table charset" 2024-03-10T00:53:36.639 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Mar 10 00:53:36 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-grafana-a[55281]: logger=migrator t=2024-03-10T00:53:36.526791381Z level=info msg="Executing migration" id="Add column external_delete_url to dashboard_snapshots table" 2024-03-10T00:53:36.639 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Mar 10 00:53:36 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-grafana-a[55281]: logger=migrator t=2024-03-10T00:53:36.527316325Z level=info msg="Executing migration" id="Add encrypted dashboard json column" 2024-03-10T00:53:36.639 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Mar 10 00:53:36 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-grafana-a[55281]: logger=migrator t=2024-03-10T00:53:36.527831951Z level=info msg="Executing migration" id="Change dashboard_encrypted column to MEDIUMBLOB" 2024-03-10T00:53:36.639 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Mar 10 00:53:36 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-grafana-a[55281]: logger=migrator t=2024-03-10T00:53:36.528083184Z level=info msg="Executing migration" id="create quota table v1" 2024-03-10T00:53:36.640 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Mar 10 00:53:36 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-grafana-a[55281]: logger=migrator t=2024-03-10T00:53:36.528461455Z level=info msg="Executing migration" id="create index UQE_quota_org_id_user_id_target - v1" 2024-03-10T00:53:36.640 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Mar 10 00:53:36 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-grafana-a[55281]: logger=migrator t=2024-03-10T00:53:36.528891797Z level=info msg="Executing migration" id="Update quota table charset" 2024-03-10T00:53:36.640 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Mar 10 00:53:36 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-grafana-a[55281]: logger=migrator t=2024-03-10T00:53:36.529145052Z level=info msg="Executing migration" id="create plugin_setting table" 2024-03-10T00:53:36.640 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Mar 10 00:53:36 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-grafana-a[55281]: logger=migrator t=2024-03-10T00:53:36.52950536Z level=info msg="Executing migration" id="create index UQE_plugin_setting_org_id_plugin_id - v1" 2024-03-10T00:53:36.640 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Mar 10 00:53:36 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-grafana-a[55281]: logger=migrator t=2024-03-10T00:53:36.529897499Z level=info msg="Executing migration" id="Add column plugin_version to plugin_settings" 2024-03-10T00:53:36.640 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Mar 10 00:53:36 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-grafana-a[55281]: logger=migrator t=2024-03-10T00:53:36.530430745Z level=info msg="Executing migration" id="Update plugin_setting table charset" 2024-03-10T00:53:36.640 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Mar 10 00:53:36 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-grafana-a[55281]: logger=migrator t=2024-03-10T00:53:36.530687453Z level=info msg="Executing migration" id="create session table" 2024-03-10T00:53:36.640 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Mar 10 00:53:36 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-grafana-a[55281]: logger=migrator t=2024-03-10T00:53:36.53106589Z level=info msg="Executing migration" id="Drop old table playlist table" 2024-03-10T00:53:36.640 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Mar 10 00:53:36 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-grafana-a[55281]: logger=migrator t=2024-03-10T00:53:36.531331535Z level=info msg="Executing migration" id="Drop old table playlist_item table" 2024-03-10T00:53:36.640 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Mar 10 00:53:36 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-grafana-a[55281]: logger=migrator t=2024-03-10T00:53:36.531582859Z level=info msg="Executing migration" id="create playlist table v2" 2024-03-10T00:53:36.640 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Mar 10 00:53:36 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-grafana-a[55281]: logger=migrator t=2024-03-10T00:53:36.531920921Z level=info msg="Executing migration" id="create playlist item table v2" 2024-03-10T00:53:36.640 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Mar 10 00:53:36 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-grafana-a[55281]: logger=migrator t=2024-03-10T00:53:36.53228536Z level=info msg="Executing migration" id="Update playlist table charset" 2024-03-10T00:53:36.641 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Mar 10 00:53:36 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-grafana-a[55281]: logger=migrator t=2024-03-10T00:53:36.532554976Z level=info msg="Executing migration" id="Update playlist_item table charset" 2024-03-10T00:53:36.641 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Mar 10 00:53:36 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-grafana-a[55281]: logger=migrator t=2024-03-10T00:53:36.532789927Z level=info msg="Executing migration" id="drop preferences table v2" 2024-03-10T00:53:36.641 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Mar 10 00:53:36 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-grafana-a[55281]: logger=migrator t=2024-03-10T00:53:36.533024467Z level=info msg="Executing migration" id="drop preferences table v3" 2024-03-10T00:53:36.641 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Mar 10 00:53:36 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-grafana-a[55281]: logger=migrator t=2024-03-10T00:53:36.533317718Z level=info msg="Executing migration" id="create preferences table v3" 2024-03-10T00:53:36.641 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Mar 10 00:53:36 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-grafana-a[55281]: logger=migrator t=2024-03-10T00:53:36.533686762Z level=info msg="Executing migration" id="Update preferences table charset" 2024-03-10T00:53:36.641 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Mar 10 00:53:36 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-grafana-a[55281]: logger=migrator t=2024-03-10T00:53:36.533919397Z level=info msg="Executing migration" id="Add column team_id in preferences" 2024-03-10T00:53:36.641 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Mar 10 00:53:36 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-grafana-a[55281]: logger=migrator t=2024-03-10T00:53:36.534472323Z level=info msg="Executing migration" id="Update team_id column values in preferences" 2024-03-10T00:53:36.641 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Mar 10 00:53:36 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-grafana-a[55281]: logger=migrator t=2024-03-10T00:53:36.534750756Z level=info msg="Executing migration" id="Add column week_start in preferences" 2024-03-10T00:53:36.641 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Mar 10 00:53:36 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-grafana-a[55281]: logger=migrator t=2024-03-10T00:53:36.535310277Z level=info msg="Executing migration" id="Add column preferences.json_data" 2024-03-10T00:53:36.641 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Mar 10 00:53:36 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-grafana-a[55281]: logger=migrator t=2024-03-10T00:53:36.535851122Z level=info msg="Executing migration" id="alter preferences.json_data to mediumtext v1" 2024-03-10T00:53:36.641 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Mar 10 00:53:36 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-grafana-a[55281]: logger=migrator t=2024-03-10T00:53:36.536090316Z level=info msg="Executing migration" id="Add preferences index org_id" 2024-03-10T00:53:36.641 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Mar 10 00:53:36 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-grafana-a[55281]: logger=migrator t=2024-03-10T00:53:36.536491174Z level=info msg="Executing migration" id="Add preferences index user_id" 2024-03-10T00:53:36.642 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Mar 10 00:53:36 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-grafana-a[55281]: logger=migrator t=2024-03-10T00:53:36.536922591Z level=info msg="Executing migration" id="create alert table v1" 2024-03-10T00:53:36.642 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Mar 10 00:53:36 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-grafana-a[55281]: logger=migrator t=2024-03-10T00:53:36.537350143Z level=info msg="Executing migration" id="add index alert org_id & id " 2024-03-10T00:53:36.642 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Mar 10 00:53:36 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-grafana-a[55281]: logger=migrator t=2024-03-10T00:53:36.537777982Z level=info msg="Executing migration" id="add index alert state" 2024-03-10T00:53:36.642 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Mar 10 00:53:36 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-grafana-a[55281]: logger=migrator t=2024-03-10T00:53:36.538233153Z level=info msg="Executing migration" id="add index alert dashboard_id" 2024-03-10T00:53:36.642 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Mar 10 00:53:36 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-grafana-a[55281]: logger=migrator t=2024-03-10T00:53:36.538625038Z level=info msg="Executing migration" id="Create alert_rule_tag table v1" 2024-03-10T00:53:36.642 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Mar 10 00:53:36 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-grafana-a[55281]: logger=migrator t=2024-03-10T00:53:36.538971911Z level=info msg="Executing migration" id="Add unique index alert_rule_tag.alert_id_tag_id" 2024-03-10T00:53:36.642 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Mar 10 00:53:36 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-grafana-a[55281]: logger=migrator t=2024-03-10T00:53:36.539381076Z level=info msg="Executing migration" id="drop index UQE_alert_rule_tag_alert_id_tag_id - v1" 2024-03-10T00:53:36.642 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Mar 10 00:53:36 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-grafana-a[55281]: logger=migrator t=2024-03-10T00:53:36.540976205Z level=info msg="Executing migration" id="Rename table alert_rule_tag to alert_rule_tag_v1 - v1" 2024-03-10T00:53:36.642 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Mar 10 00:53:36 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-grafana-a[55281]: logger=migrator t=2024-03-10T00:53:36.542507372Z level=info msg="Executing migration" id="Create alert_rule_tag table v2" 2024-03-10T00:53:36.643 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Mar 10 00:53:36 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-grafana-a[55281]: logger=migrator t=2024-03-10T00:53:36.54289354Z level=info msg="Executing migration" id="create index UQE_alert_rule_tag_alert_id_tag_id - Add unique index alert_rule_tag.alert_id_tag_id V2" 2024-03-10T00:53:36.643 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Mar 10 00:53:36 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-grafana-a[55281]: logger=migrator t=2024-03-10T00:53:36.543306451Z level=info msg="Executing migration" id="copy alert_rule_tag v1 to v2" 2024-03-10T00:53:36.643 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Mar 10 00:53:36 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-grafana-a[55281]: logger=migrator t=2024-03-10T00:53:36.543600075Z level=info msg="Executing migration" id="drop table alert_rule_tag_v1" 2024-03-10T00:53:36.643 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Mar 10 00:53:36 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-grafana-a[55281]: logger=migrator t=2024-03-10T00:53:36.543903441Z level=info msg="Executing migration" id="create alert_notification table v1" 2024-03-10T00:53:36.643 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Mar 10 00:53:36 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-grafana-a[55281]: logger=migrator t=2024-03-10T00:53:36.544269004Z level=info msg="Executing migration" id="Add column is_default" 2024-03-10T00:53:36.643 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Mar 10 00:53:36 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-grafana-a[55281]: logger=migrator t=2024-03-10T00:53:36.544847152Z level=info msg="Executing migration" id="Add column frequency" 2024-03-10T00:53:36.643 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Mar 10 00:53:36 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-grafana-a[55281]: logger=migrator t=2024-03-10T00:53:36.545439358Z level=info msg="Executing migration" id="Add column send_reminder" 2024-03-10T00:53:36.643 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Mar 10 00:53:36 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-grafana-a[55281]: logger=migrator t=2024-03-10T00:53:36.546017027Z level=info msg="Executing migration" id="Add column disable_resolve_message" 2024-03-10T00:53:36.643 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Mar 10 00:53:36 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-grafana-a[55281]: logger=migrator t=2024-03-10T00:53:36.546634054Z level=info msg="Executing migration" id="add index alert_notification org_id & name" 2024-03-10T00:53:36.643 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Mar 10 00:53:36 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-grafana-a[55281]: logger=migrator t=2024-03-10T00:53:36.547038495Z level=info msg="Executing migration" id="Update alert table charset" 2024-03-10T00:53:36.644 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Mar 10 00:53:36 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-grafana-a[55281]: logger=migrator t=2024-03-10T00:53:36.547344338Z level=info msg="Executing migration" id="Update alert_notification table charset" 2024-03-10T00:53:36.644 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Mar 10 00:53:36 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-grafana-a[55281]: logger=migrator t=2024-03-10T00:53:36.547604709Z level=info msg="Executing migration" id="create notification_journal table v1" 2024-03-10T00:53:36.644 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Mar 10 00:53:36 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-grafana-a[55281]: logger=migrator t=2024-03-10T00:53:36.547960537Z level=info msg="Executing migration" id="add index notification_journal org_id & alert_id & notifier_id" 2024-03-10T00:53:36.644 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Mar 10 00:53:36 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-grafana-a[55281]: logger=migrator t=2024-03-10T00:53:36.548377777Z level=info msg="Executing migration" id="drop alert_notification_journal" 2024-03-10T00:53:36.644 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Mar 10 00:53:36 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-grafana-a[55281]: logger=migrator t=2024-03-10T00:53:36.548732615Z level=info msg="Executing migration" id="create alert_notification_state table v1" 2024-03-10T00:53:36.644 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Mar 10 00:53:36 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-grafana-a[55281]: logger=migrator t=2024-03-10T00:53:36.549059382Z level=info msg="Executing migration" id="add index alert_notification_state org_id & alert_id & notifier_id" 2024-03-10T00:53:36.644 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Mar 10 00:53:36 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-grafana-a[55281]: logger=migrator t=2024-03-10T00:53:36.549466752Z level=info msg="Executing migration" id="Add for to alert table" 2024-03-10T00:53:36.644 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Mar 10 00:53:36 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-grafana-a[55281]: logger=migrator t=2024-03-10T00:53:36.550037168Z level=info msg="Executing migration" id="Add column uid in alert_notification" 2024-03-10T00:53:36.644 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Mar 10 00:53:36 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-grafana-a[55281]: logger=migrator t=2024-03-10T00:53:36.550639376Z level=info msg="Executing migration" id="Update uid column values in alert_notification" 2024-03-10T00:53:36.644 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Mar 10 00:53:36 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-grafana-a[55281]: logger=migrator t=2024-03-10T00:53:36.550933018Z level=info msg="Executing migration" id="Add unique index alert_notification_org_id_uid" 2024-03-10T00:53:36.644 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Mar 10 00:53:36 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-grafana-a[55281]: logger=migrator t=2024-03-10T00:53:36.551370364Z level=info msg="Executing migration" id="Remove unique index org_id_name" 2024-03-10T00:53:36.644 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Mar 10 00:53:36 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-grafana-a[55281]: logger=migrator t=2024-03-10T00:53:36.551750944Z level=info msg="Executing migration" id="Add column secure_settings in alert_notification" 2024-03-10T00:53:36.644 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Mar 10 00:53:36 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-grafana-a[55281]: logger=migrator t=2024-03-10T00:53:36.552371409Z level=info msg="Executing migration" id="alter alert.settings to mediumtext" 2024-03-10T00:53:36.645 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Mar 10 00:53:36 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-grafana-a[55281]: logger=migrator t=2024-03-10T00:53:36.552642892Z level=info msg="Executing migration" id="Add non-unique index alert_notification_state_alert_id" 2024-03-10T00:53:36.645 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Mar 10 00:53:36 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-grafana-a[55281]: logger=migrator t=2024-03-10T00:53:36.553011356Z level=info msg="Executing migration" id="Add non-unique index alert_rule_tag_alert_id" 2024-03-10T00:53:36.645 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Mar 10 00:53:36 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-grafana-a[55281]: logger=migrator t=2024-03-10T00:53:36.553401555Z level=info msg="Executing migration" id="Drop old annotation table v4" 2024-03-10T00:53:36.645 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Mar 10 00:53:36 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-grafana-a[55281]: logger=migrator t=2024-03-10T00:53:36.553688209Z level=info msg="Executing migration" id="create annotation table v5" 2024-03-10T00:53:36.645 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Mar 10 00:53:36 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-grafana-a[55281]: logger=migrator t=2024-03-10T00:53:36.554081565Z level=info msg="Executing migration" id="add index annotation 0 v3" 2024-03-10T00:53:36.645 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Mar 10 00:53:36 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-grafana-a[55281]: logger=migrator t=2024-03-10T00:53:36.55450435Z level=info msg="Executing migration" id="add index annotation 1 v3" 2024-03-10T00:53:36.645 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Mar 10 00:53:36 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-grafana-a[55281]: logger=migrator t=2024-03-10T00:53:36.554910813Z level=info msg="Executing migration" id="add index annotation 2 v3" 2024-03-10T00:53:36.645 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Mar 10 00:53:36 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-grafana-a[55281]: logger=migrator t=2024-03-10T00:53:36.555360857Z level=info msg="Executing migration" id="add index annotation 3 v3" 2024-03-10T00:53:36.645 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Mar 10 00:53:36 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-grafana-a[55281]: logger=migrator t=2024-03-10T00:53:36.555784598Z level=info msg="Executing migration" id="add index annotation 4 v3" 2024-03-10T00:53:36.645 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Mar 10 00:53:36 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-grafana-a[55281]: logger=migrator t=2024-03-10T00:53:36.556219166Z level=info msg="Executing migration" id="Update annotation table charset" 2024-03-10T00:53:36.645 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Mar 10 00:53:36 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-grafana-a[55281]: logger=migrator t=2024-03-10T00:53:36.556487909Z level=info msg="Executing migration" id="Add column region_id to annotation table" 2024-03-10T00:53:36.645 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Mar 10 00:53:36 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-grafana-a[55281]: logger=migrator t=2024-03-10T00:53:36.557102707Z level=info msg="Executing migration" id="Drop category_id index" 2024-03-10T00:53:36.645 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Mar 10 00:53:36 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-grafana-a[55281]: logger=migrator t=2024-03-10T00:53:36.557512019Z level=info msg="Executing migration" id="Add column tags to annotation table" 2024-03-10T00:53:36.645 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Mar 10 00:53:36 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-grafana-a[55281]: logger=migrator t=2024-03-10T00:53:36.558101178Z level=info msg="Executing migration" id="Create annotation_tag table v2" 2024-03-10T00:53:36.646 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Mar 10 00:53:36 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-grafana-a[55281]: logger=migrator t=2024-03-10T00:53:36.5584753Z level=info msg="Executing migration" id="Add unique index annotation_tag.annotation_id_tag_id" 2024-03-10T00:53:36.646 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Mar 10 00:53:36 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-grafana-a[55281]: logger=migrator t=2024-03-10T00:53:36.558874766Z level=info msg="Executing migration" id="drop index UQE_annotation_tag_annotation_id_tag_id - v2" 2024-03-10T00:53:36.646 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Mar 10 00:53:36 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-grafana-a[55281]: logger=migrator t=2024-03-10T00:53:36.559294497Z level=info msg="Executing migration" id="Rename table annotation_tag to annotation_tag_v2 - v2" 2024-03-10T00:53:36.646 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Mar 10 00:53:36 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-grafana-a[55281]: logger=migrator t=2024-03-10T00:53:36.560813034Z level=info msg="Executing migration" id="Create annotation_tag table v3" 2024-03-10T00:53:36.646 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Mar 10 00:53:36 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-grafana-a[55281]: logger=migrator t=2024-03-10T00:53:36.561174368Z level=info msg="Executing migration" id="create index UQE_annotation_tag_annotation_id_tag_id - Add unique index annotation_tag.annotation_id_tag_id V3" 2024-03-10T00:53:36.646 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Mar 10 00:53:36 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-grafana-a[55281]: logger=migrator t=2024-03-10T00:53:36.561589838Z level=info msg="Executing migration" id="copy annotation_tag v2 to v3" 2024-03-10T00:53:36.646 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Mar 10 00:53:36 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-grafana-a[55281]: logger=migrator t=2024-03-10T00:53:36.561862638Z level=info msg="Executing migration" id="drop table annotation_tag_v2" 2024-03-10T00:53:36.646 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Mar 10 00:53:36 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-grafana-a[55281]: logger=migrator t=2024-03-10T00:53:36.562164738Z level=info msg="Executing migration" id="Update alert annotations and set TEXT to empty" 2024-03-10T00:53:36.646 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Mar 10 00:53:36 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-grafana-a[55281]: logger=migrator t=2024-03-10T00:53:36.562447565Z level=info msg="Executing migration" id="Add created time to annotation table" 2024-03-10T00:53:36.646 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Mar 10 00:53:36 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-grafana-a[55281]: logger=migrator t=2024-03-10T00:53:36.563043291Z level=info msg="Executing migration" id="Add updated time to annotation table" 2024-03-10T00:53:36.646 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Mar 10 00:53:36 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-grafana-a[55281]: logger=migrator t=2024-03-10T00:53:36.563702236Z level=info msg="Executing migration" id="Add index for created in annotation table" 2024-03-10T00:53:36.646 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Mar 10 00:53:36 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-grafana-a[55281]: logger=migrator t=2024-03-10T00:53:36.564077679Z level=info msg="Executing migration" id="Add index for updated in annotation table" 2024-03-10T00:53:36.646 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Mar 10 00:53:36 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-grafana-a[55281]: logger=migrator t=2024-03-10T00:53:36.564473305Z level=info msg="Executing migration" id="Convert existing annotations from seconds to milliseconds" 2024-03-10T00:53:36.647 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Mar 10 00:53:36 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-grafana-a[55281]: logger=migrator t=2024-03-10T00:53:36.564729645Z level=info msg="Executing migration" id="Add epoch_end column" 2024-03-10T00:53:36.647 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Mar 10 00:53:36 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-grafana-a[55281]: logger=migrator t=2024-03-10T00:53:36.565397518Z level=info msg="Executing migration" id="Add index for epoch_end" 2024-03-10T00:53:36.647 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Mar 10 00:53:36 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-grafana-a[55281]: logger=migrator t=2024-03-10T00:53:36.565802771Z level=info msg="Executing migration" id="Make epoch_end the same as epoch" 2024-03-10T00:53:36.647 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Mar 10 00:53:36 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-grafana-a[55281]: logger=migrator t=2024-03-10T00:53:36.566060504Z level=info msg="Executing migration" id="Move region to single row" 2024-03-10T00:53:36.647 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Mar 10 00:53:36 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-grafana-a[55281]: logger=migrator t=2024-03-10T00:53:36.566416309Z level=info msg="Executing migration" id="Remove index org_id_epoch from annotation table" 2024-03-10T00:53:36.647 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Mar 10 00:53:36 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-grafana-a[55281]: logger=migrator t=2024-03-10T00:53:36.56677247Z level=info msg="Executing migration" id="Remove index org_id_dashboard_id_panel_id_epoch from annotation table" 2024-03-10T00:53:36.647 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Mar 10 00:53:36 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-grafana-a[55281]: logger=migrator t=2024-03-10T00:53:36.567141943Z level=info msg="Executing migration" id="Add index for org_id_dashboard_id_epoch_end_epoch on annotation table" 2024-03-10T00:53:36.647 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Mar 10 00:53:36 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-grafana-a[55281]: logger=migrator t=2024-03-10T00:53:36.567548606Z level=info msg="Executing migration" id="Add index for org_id_epoch_end_epoch on annotation table" 2024-03-10T00:53:36.647 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Mar 10 00:53:36 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-grafana-a[55281]: logger=migrator t=2024-03-10T00:53:36.567919074Z level=info msg="Executing migration" id="Remove index org_id_epoch_epoch_end from annotation table" 2024-03-10T00:53:36.647 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Mar 10 00:53:36 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-grafana-a[55281]: logger=migrator t=2024-03-10T00:53:36.568307256Z level=info msg="Executing migration" id="Add index for alert_id on annotation table" 2024-03-10T00:53:36.647 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Mar 10 00:53:36 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-grafana-a[55281]: logger=migrator t=2024-03-10T00:53:36.568696145Z level=info msg="Executing migration" id="Increase tags column to length 4096" 2024-03-10T00:53:36.647 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Mar 10 00:53:36 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-grafana-a[55281]: logger=migrator t=2024-03-10T00:53:36.568954073Z level=info msg="Executing migration" id="create test_data table" 2024-03-10T00:53:36.647 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Mar 10 00:53:36 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-grafana-a[55281]: logger=migrator t=2024-03-10T00:53:36.569350481Z level=info msg="Executing migration" id="create dashboard_version table v1" 2024-03-10T00:53:36.648 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Mar 10 00:53:36 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-grafana-a[55281]: logger=migrator t=2024-03-10T00:53:36.569735096Z level=info msg="Executing migration" id="add index dashboard_version.dashboard_id" 2024-03-10T00:53:36.648 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Mar 10 00:53:36 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-grafana-a[55281]: logger=migrator t=2024-03-10T00:53:36.570157277Z level=info msg="Executing migration" id="add unique index dashboard_version.dashboard_id and dashboard_version.version" 2024-03-10T00:53:36.648 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Mar 10 00:53:36 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-grafana-a[55281]: logger=migrator t=2024-03-10T00:53:36.570550221Z level=info msg="Executing migration" id="Set dashboard version to 1 where 0" 2024-03-10T00:53:36.648 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Mar 10 00:53:36 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-grafana-a[55281]: logger=migrator t=2024-03-10T00:53:36.570804779Z level=info msg="Executing migration" id="save existing dashboard data in dashboard_version table v1" 2024-03-10T00:53:36.648 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Mar 10 00:53:36 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-grafana-a[55281]: logger=migrator t=2024-03-10T00:53:36.571069547Z level=info msg="Executing migration" id="alter dashboard_version.data to mediumtext v1" 2024-03-10T00:53:36.648 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Mar 10 00:53:36 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-grafana-a[55281]: logger=migrator t=2024-03-10T00:53:36.571383087Z level=info msg="Executing migration" id="create team table" 2024-03-10T00:53:36.648 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Mar 10 00:53:36 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-grafana-a[55281]: logger=migrator t=2024-03-10T00:53:36.571733961Z level=info msg="Executing migration" id="add index team.org_id" 2024-03-10T00:53:36.648 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Mar 10 00:53:36 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-grafana-a[55281]: logger=migrator t=2024-03-10T00:53:36.572163325Z level=info msg="Executing migration" id="add unique index team_org_id_name" 2024-03-10T00:53:36.648 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Mar 10 00:53:36 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-grafana-a[55281]: logger=migrator t=2024-03-10T00:53:36.572563707Z level=info msg="Executing migration" id="create team member table" 2024-03-10T00:53:36.648 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Mar 10 00:53:36 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-grafana-a[55281]: logger=migrator t=2024-03-10T00:53:36.572920844Z level=info msg="Executing migration" id="add index team_member.org_id" 2024-03-10T00:53:36.648 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Mar 10 00:53:36 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-grafana-a[55281]: logger=migrator t=2024-03-10T00:53:36.573332377Z level=info msg="Executing migration" id="add unique index team_member_org_id_team_id_user_id" 2024-03-10T00:53:36.648 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Mar 10 00:53:36 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-grafana-a[55281]: logger=migrator t=2024-03-10T00:53:36.573731608Z level=info msg="Executing migration" id="add index team_member.team_id" 2024-03-10T00:53:36.648 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Mar 10 00:53:36 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-grafana-a[55281]: logger=migrator t=2024-03-10T00:53:36.574154498Z level=info msg="Executing migration" id="Add column email to team table" 2024-03-10T00:53:36.648 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Mar 10 00:53:36 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-grafana-a[55281]: logger=migrator t=2024-03-10T00:53:36.575100544Z level=info msg="Executing migration" id="Add column external to team_member table" 2024-03-10T00:53:36.649 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Mar 10 00:53:36 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-grafana-a[55281]: logger=migrator t=2024-03-10T00:53:36.575781473Z level=info msg="Executing migration" id="Add column permission to team_member table" 2024-03-10T00:53:36.649 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Mar 10 00:53:36 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-grafana-a[55281]: logger=migrator t=2024-03-10T00:53:36.576461042Z level=info msg="Executing migration" id="create dashboard acl table" 2024-03-10T00:53:36.649 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Mar 10 00:53:36 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-grafana-a[55281]: logger=migrator t=2024-03-10T00:53:36.57682975Z level=info msg="Executing migration" id="add index dashboard_acl_dashboard_id" 2024-03-10T00:53:36.649 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Mar 10 00:53:36 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-grafana-a[55281]: logger=migrator t=2024-03-10T00:53:36.577273401Z level=info msg="Executing migration" id="add unique index dashboard_acl_dashboard_id_user_id" 2024-03-10T00:53:36.649 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Mar 10 00:53:36 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-grafana-a[55281]: logger=migrator t=2024-03-10T00:53:36.577682471Z level=info msg="Executing migration" id="add unique index dashboard_acl_dashboard_id_team_id" 2024-03-10T00:53:36.649 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Mar 10 00:53:36 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-grafana-a[55281]: logger=migrator t=2024-03-10T00:53:36.578073553Z level=info msg="Executing migration" id="add index dashboard_acl_user_id" 2024-03-10T00:53:36.649 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Mar 10 00:53:36 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-grafana-a[55281]: logger=migrator t=2024-03-10T00:53:36.578576453Z level=info msg="Executing migration" id="add index dashboard_acl_team_id" 2024-03-10T00:53:36.649 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Mar 10 00:53:36 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-grafana-a[55281]: logger=migrator t=2024-03-10T00:53:36.578972984Z level=info msg="Executing migration" id="add index dashboard_acl_org_id_role" 2024-03-10T00:53:36.649 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Mar 10 00:53:36 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-grafana-a[55281]: logger=migrator t=2024-03-10T00:53:36.579381837Z level=info msg="Executing migration" id="add index dashboard_permission" 2024-03-10T00:53:36.649 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Mar 10 00:53:36 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-grafana-a[55281]: logger=migrator t=2024-03-10T00:53:36.579776841Z level=info msg="Executing migration" id="save default acl rules in dashboard_acl table" 2024-03-10T00:53:36.649 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Mar 10 00:53:36 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-grafana-a[55281]: logger=migrator t=2024-03-10T00:53:36.580133313Z level=info msg="Executing migration" id="delete acl rules for deleted dashboards and folders" 2024-03-10T00:53:36.649 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Mar 10 00:53:36 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-grafana-a[55281]: logger=migrator t=2024-03-10T00:53:36.580424065Z level=info msg="Executing migration" id="create tag table" 2024-03-10T00:53:36.649 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Mar 10 00:53:36 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-grafana-a[55281]: logger=migrator t=2024-03-10T00:53:36.580784401Z level=info msg="Executing migration" id="add index tag.key_value" 2024-03-10T00:53:36.649 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Mar 10 00:53:36 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-grafana-a[55281]: logger=migrator t=2024-03-10T00:53:36.581195638Z level=info msg="Executing migration" id="create login attempt table" 2024-03-10T00:53:36.650 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Mar 10 00:53:36 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-grafana-a[55281]: logger=migrator t=2024-03-10T00:53:36.581579723Z level=info msg="Executing migration" id="add index login_attempt.username" 2024-03-10T00:53:36.650 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Mar 10 00:53:36 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-grafana-a[55281]: logger=migrator t=2024-03-10T00:53:36.581973421Z level=info msg="Executing migration" id="drop index IDX_login_attempt_username - v1" 2024-03-10T00:53:36.650 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Mar 10 00:53:36 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-grafana-a[55281]: logger=migrator t=2024-03-10T00:53:36.582389433Z level=info msg="Executing migration" id="Rename table login_attempt to login_attempt_tmp_qwerty - v1" 2024-03-10T00:53:36.650 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Mar 10 00:53:36 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-grafana-a[55281]: logger=migrator t=2024-03-10T00:53:36.584376149Z level=info msg="Executing migration" id="create login_attempt v2" 2024-03-10T00:53:36.650 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Mar 10 00:53:36 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-grafana-a[55281]: logger=migrator t=2024-03-10T00:53:36.58477126Z level=info msg="Executing migration" id="create index IDX_login_attempt_username - v2" 2024-03-10T00:53:36.650 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Mar 10 00:53:36 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-grafana-a[55281]: logger=migrator t=2024-03-10T00:53:36.585173801Z level=info msg="Executing migration" id="copy login_attempt v1 to v2" 2024-03-10T00:53:36.650 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Mar 10 00:53:36 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-grafana-a[55281]: logger=migrator t=2024-03-10T00:53:36.585476996Z level=info msg="Executing migration" id="drop login_attempt_tmp_qwerty" 2024-03-10T00:53:36.650 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Mar 10 00:53:36 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-grafana-a[55281]: logger=migrator t=2024-03-10T00:53:36.58577713Z level=info msg="Executing migration" id="create user auth table" 2024-03-10T00:53:36.650 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Mar 10 00:53:36 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-grafana-a[55281]: logger=migrator t=2024-03-10T00:53:36.586102105Z level=info msg="Executing migration" id="create index IDX_user_auth_auth_module_auth_id - v1" 2024-03-10T00:53:36.650 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Mar 10 00:53:36 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-grafana-a[55281]: logger=migrator t=2024-03-10T00:53:36.586529364Z level=info msg="Executing migration" id="alter user_auth.auth_id to length 190" 2024-03-10T00:53:36.650 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Mar 10 00:53:36 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-grafana-a[55281]: logger=migrator t=2024-03-10T00:53:36.586764749Z level=info msg="Executing migration" id="Add OAuth access token to user_auth" 2024-03-10T00:53:36.650 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Mar 10 00:53:36 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-grafana-a[55281]: logger=migrator t=2024-03-10T00:53:36.587475697Z level=info msg="Executing migration" id="Add OAuth refresh token to user_auth" 2024-03-10T00:53:36.650 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Mar 10 00:53:36 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-grafana-a[55281]: logger=migrator t=2024-03-10T00:53:36.588196799Z level=info msg="Executing migration" id="Add OAuth token type to user_auth" 2024-03-10T00:53:36.650 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Mar 10 00:53:36 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-grafana-a[55281]: logger=migrator t=2024-03-10T00:53:36.588942652Z level=info msg="Executing migration" id="Add OAuth expiry to user_auth" 2024-03-10T00:53:36.651 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Mar 10 00:53:36 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-grafana-a[55281]: logger=migrator t=2024-03-10T00:53:36.589688878Z level=info msg="Executing migration" id="Add index to user_id column in user_auth" 2024-03-10T00:53:36.651 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Mar 10 00:53:36 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-grafana-a[55281]: logger=migrator t=2024-03-10T00:53:36.590130529Z level=info msg="Executing migration" id="Add OAuth ID token to user_auth" 2024-03-10T00:53:36.651 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Mar 10 00:53:36 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-grafana-a[55281]: logger=migrator t=2024-03-10T00:53:36.590883328Z level=info msg="Executing migration" id="create server_lock table" 2024-03-10T00:53:36.651 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Mar 10 00:53:36 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-grafana-a[55281]: logger=migrator t=2024-03-10T00:53:36.591280269Z level=info msg="Executing migration" id="add index server_lock.operation_uid" 2024-03-10T00:53:36.651 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Mar 10 00:53:36 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-grafana-a[55281]: logger=migrator t=2024-03-10T00:53:36.591705674Z level=info msg="Executing migration" id="create user auth token table" 2024-03-10T00:53:36.651 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Mar 10 00:53:36 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-grafana-a[55281]: logger=migrator t=2024-03-10T00:53:36.592125771Z level=info msg="Executing migration" id="add unique index user_auth_token.auth_token" 2024-03-10T00:53:36.651 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Mar 10 00:53:36 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-grafana-a[55281]: logger=migrator t=2024-03-10T00:53:36.592540994Z level=info msg="Executing migration" id="add unique index user_auth_token.prev_auth_token" 2024-03-10T00:53:36.651 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Mar 10 00:53:36 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-grafana-a[55281]: logger=migrator t=2024-03-10T00:53:36.592950745Z level=info msg="Executing migration" id="add index user_auth_token.user_id" 2024-03-10T00:53:36.651 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Mar 10 00:53:36 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-grafana-a[55281]: logger=migrator t=2024-03-10T00:53:36.593372905Z level=info msg="Executing migration" id="Add revoked_at to the user auth token" 2024-03-10T00:53:36.651 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Mar 10 00:53:36 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-grafana-a[55281]: logger=migrator t=2024-03-10T00:53:36.594126873Z level=info msg="Executing migration" id="create cache_data table" 2024-03-10T00:53:36.651 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Mar 10 00:53:36 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-grafana-a[55281]: logger=migrator t=2024-03-10T00:53:36.594554331Z level=info msg="Executing migration" id="add unique index cache_data.cache_key" 2024-03-10T00:53:36.651 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Mar 10 00:53:36 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-grafana-a[55281]: logger=migrator t=2024-03-10T00:53:36.594970332Z level=info msg="Executing migration" id="create short_url table v1" 2024-03-10T00:53:36.651 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Mar 10 00:53:36 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-grafana-a[55281]: logger=migrator t=2024-03-10T00:53:36.595391151Z level=info msg="Executing migration" id="add index short_url.org_id-uid" 2024-03-10T00:53:36.651 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Mar 10 00:53:36 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-grafana-a[55281]: logger=migrator t=2024-03-10T00:53:36.595803546Z level=info msg="Executing migration" id="alter table short_url alter column created_by type to bigint" 2024-03-10T00:53:36.652 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Mar 10 00:53:36 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-grafana-a[55281]: logger=migrator t=2024-03-10T00:53:36.596066447Z level=info msg="Executing migration" id="delete alert_definition table" 2024-03-10T00:53:36.652 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Mar 10 00:53:36 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-grafana-a[55281]: logger=migrator t=2024-03-10T00:53:36.596370059Z level=info msg="Executing migration" id="recreate alert_definition table" 2024-03-10T00:53:36.652 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Mar 10 00:53:36 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-grafana-a[55281]: logger=migrator t=2024-03-10T00:53:36.596824522Z level=info msg="Executing migration" id="add index in alert_definition on org_id and title columns" 2024-03-10T00:53:36.652 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Mar 10 00:53:36 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-grafana-a[55281]: logger=migrator t=2024-03-10T00:53:36.597261943Z level=info msg="Executing migration" id="add index in alert_definition on org_id and uid columns" 2024-03-10T00:53:36.652 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Mar 10 00:53:36 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-grafana-a[55281]: logger=migrator t=2024-03-10T00:53:36.597676501Z level=info msg="Executing migration" id="alter alert_definition table data column to mediumtext in mysql" 2024-03-10T00:53:36.652 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Mar 10 00:53:36 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-grafana-a[55281]: logger=migrator t=2024-03-10T00:53:36.597935226Z level=info msg="Executing migration" id="drop index in alert_definition on org_id and title columns" 2024-03-10T00:53:36.652 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Mar 10 00:53:36 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-grafana-a[55281]: logger=migrator t=2024-03-10T00:53:36.598347534Z level=info msg="Executing migration" id="drop index in alert_definition on org_id and uid columns" 2024-03-10T00:53:36.652 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Mar 10 00:53:36 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-grafana-a[55281]: logger=migrator t=2024-03-10T00:53:36.598787359Z level=info msg="Executing migration" id="add unique index in alert_definition on org_id and title columns" 2024-03-10T00:53:36.652 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Mar 10 00:53:36 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-grafana-a[55281]: logger=migrator t=2024-03-10T00:53:36.599192809Z level=info msg="Executing migration" id="add unique index in alert_definition on org_id and uid columns" 2024-03-10T00:53:36.653 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Mar 10 00:53:36 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-grafana-a[55281]: logger=migrator t=2024-03-10T00:53:36.599632092Z level=info msg="Executing migration" id="Add column paused in alert_definition" 2024-03-10T00:53:36.653 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Mar 10 00:53:36 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-grafana-a[55281]: logger=migrator t=2024-03-10T00:53:36.600406152Z level=info msg="Executing migration" id="drop alert_definition table" 2024-03-10T00:53:36.653 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Mar 10 00:53:36 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-grafana-a[55281]: logger=migrator t=2024-03-10T00:53:36.600819879Z level=info msg="Executing migration" id="delete alert_definition_version table" 2024-03-10T00:53:36.653 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Mar 10 00:53:36 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-grafana-a[55281]: logger=migrator t=2024-03-10T00:53:36.601083098Z level=info msg="Executing migration" id="recreate alert_definition_version table" 2024-03-10T00:53:36.653 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Mar 10 00:53:36 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-grafana-a[55281]: logger=migrator t=2024-03-10T00:53:36.601513214Z level=info msg="Executing migration" id="add index in alert_definition_version table on alert_definition_id and version columns" 2024-03-10T00:53:36.653 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Mar 10 00:53:36 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-grafana-a[55281]: logger=migrator t=2024-03-10T00:53:36.601904804Z level=info msg="Executing migration" id="add index in alert_definition_version table on alert_definition_uid and version columns" 2024-03-10T00:53:36.653 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Mar 10 00:53:36 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-grafana-a[55281]: logger=migrator t=2024-03-10T00:53:36.602348304Z level=info msg="Executing migration" id="alter alert_definition_version table data column to mediumtext in mysql" 2024-03-10T00:53:36.653 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Mar 10 00:53:36 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-grafana-a[55281]: logger=migrator t=2024-03-10T00:53:36.602636901Z level=info msg="Executing migration" id="drop alert_definition_version table" 2024-03-10T00:53:36.653 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Mar 10 00:53:36 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-grafana-a[55281]: logger=migrator t=2024-03-10T00:53:36.603016897Z level=info msg="Executing migration" id="create alert_instance table" 2024-03-10T00:53:36.653 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Mar 10 00:53:36 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-grafana-a[55281]: logger=migrator t=2024-03-10T00:53:36.603427583Z level=info msg="Executing migration" id="add index in alert_instance table on def_org_id, def_uid and current_state columns" 2024-03-10T00:53:36.653 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Mar 10 00:53:36 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-grafana-a[55281]: logger=migrator t=2024-03-10T00:53:36.603863071Z level=info msg="Executing migration" id="add index in alert_instance table on def_org_id, current_state columns" 2024-03-10T00:53:36.653 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Mar 10 00:53:36 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-grafana-a[55281]: logger=migrator t=2024-03-10T00:53:36.604319675Z level=info msg="Executing migration" id="add column current_state_end to alert_instance" 2024-03-10T00:53:36.653 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Mar 10 00:53:36 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-grafana-a[55281]: logger=migrator t=2024-03-10T00:53:36.605134023Z level=info msg="Executing migration" id="remove index def_org_id, def_uid, current_state on alert_instance" 2024-03-10T00:53:36.654 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Mar 10 00:53:36 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-grafana-a[55281]: logger=migrator t=2024-03-10T00:53:36.605563347Z level=info msg="Executing migration" id="remove index def_org_id, current_state on alert_instance" 2024-03-10T00:53:36.654 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Mar 10 00:53:36 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-grafana-a[55281]: logger=migrator t=2024-03-10T00:53:36.605932286Z level=info msg="Executing migration" id="rename def_org_id to rule_org_id in alert_instance" 2024-03-10T00:53:36.654 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Mar 10 00:53:36 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-grafana-a[55281]: logger=migrator t=2024-03-10T00:53:36.609220655Z level=info msg="Executing migration" id="rename def_uid to rule_uid in alert_instance" 2024-03-10T00:53:36.654 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Mar 10 00:53:36 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-grafana-a[55281]: logger=migrator t=2024-03-10T00:53:36.61263975Z level=info msg="Executing migration" id="add index rule_org_id, rule_uid, current_state on alert_instance" 2024-03-10T00:53:36.654 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Mar 10 00:53:36 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-grafana-a[55281]: logger=migrator t=2024-03-10T00:53:36.613054906Z level=info msg="Executing migration" id="add index rule_org_id, current_state on alert_instance" 2024-03-10T00:53:36.654 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Mar 10 00:53:36 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-grafana-a[55281]: logger=migrator t=2024-03-10T00:53:36.613514006Z level=info msg="Executing migration" id="add current_reason column related to current_state" 2024-03-10T00:53:36.654 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Mar 10 00:53:36 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-grafana-a[55281]: logger=migrator t=2024-03-10T00:53:36.614381653Z level=info msg="Executing migration" id="create alert_rule table" 2024-03-10T00:53:36.654 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Mar 10 00:53:36 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-grafana-a[55281]: logger=migrator t=2024-03-10T00:53:36.614786775Z level=info msg="Executing migration" id="add index in alert_rule on org_id and title columns" 2024-03-10T00:53:36.654 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Mar 10 00:53:36 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-grafana-a[55281]: logger=migrator t=2024-03-10T00:53:36.615218529Z level=info msg="Executing migration" id="add index in alert_rule on org_id and uid columns" 2024-03-10T00:53:36.654 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Mar 10 00:53:36 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-grafana-a[55281]: logger=migrator t=2024-03-10T00:53:36.615668347Z level=info msg="Executing migration" id="add index in alert_rule on org_id, namespace_uid, group_uid columns" 2024-03-10T00:53:36.655 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Mar 10 00:53:36 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-grafana-a[55281]: logger=migrator t=2024-03-10T00:53:36.616108147Z level=info msg="Executing migration" id="alter alert_rule table data column to mediumtext in mysql" 2024-03-10T00:53:36.655 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Mar 10 00:53:36 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-grafana-a[55281]: logger=migrator t=2024-03-10T00:53:36.616430549Z level=info msg="Executing migration" id="add column for to alert_rule" 2024-03-10T00:53:36.655 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Mar 10 00:53:36 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-grafana-a[55281]: logger=migrator t=2024-03-10T00:53:36.617273746Z level=info msg="Executing migration" id="add column annotations to alert_rule" 2024-03-10T00:53:36.655 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Mar 10 00:53:36 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-grafana-a[55281]: logger=migrator t=2024-03-10T00:53:36.618081164Z level=info msg="Executing migration" id="add column labels to alert_rule" 2024-03-10T00:53:36.655 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Mar 10 00:53:36 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-grafana-a[55281]: logger=migrator t=2024-03-10T00:53:36.618960066Z level=info msg="Executing migration" id="remove unique index from alert_rule on org_id, title columns" 2024-03-10T00:53:36.655 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Mar 10 00:53:36 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-grafana-a[55281]: logger=migrator t=2024-03-10T00:53:36.619379324Z level=info msg="Executing migration" id="add index in alert_rule on org_id, namespase_uid and title columns" 2024-03-10T00:53:36.655 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Mar 10 00:53:36 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-grafana-a[55281]: logger=migrator t=2024-03-10T00:53:36.61981688Z level=info msg="Executing migration" id="add dashboard_uid column to alert_rule" 2024-03-10T00:53:36.655 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Mar 10 00:53:36 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-grafana-a[55281]: logger=migrator t=2024-03-10T00:53:36.620674483Z level=info msg="Executing migration" id="add panel_id column to alert_rule" 2024-03-10T00:53:36.655 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Mar 10 00:53:36 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-grafana-a[55281]: logger=migrator t=2024-03-10T00:53:36.621560201Z level=info msg="Executing migration" id="add index in alert_rule on org_id, dashboard_uid and panel_id columns" 2024-03-10T00:53:36.655 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Mar 10 00:53:36 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-grafana-a[55281]: logger=migrator t=2024-03-10T00:53:36.622010167Z level=info msg="Executing migration" id="add rule_group_idx column to alert_rule" 2024-03-10T00:53:36.655 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Mar 10 00:53:36 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-grafana-a[55281]: logger=migrator t=2024-03-10T00:53:36.622845379Z level=info msg="Executing migration" id="add is_paused column to alert_rule table" 2024-03-10T00:53:36.655 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Mar 10 00:53:36 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-grafana-a[55281]: logger=migrator t=2024-03-10T00:53:36.623743471Z level=info msg="Executing migration" id="fix is_paused column for alert_rule table" 2024-03-10T00:53:36.655 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Mar 10 00:53:36 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-grafana-a[55281]: logger=migrator t=2024-03-10T00:53:36.624054644Z level=info msg="Executing migration" id="create alert_rule_version table" 2024-03-10T00:53:36.656 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Mar 10 00:53:36 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-grafana-a[55281]: logger=migrator t=2024-03-10T00:53:36.62453619Z level=info msg="Executing migration" id="add index in alert_rule_version table on rule_org_id, rule_uid and version columns" 2024-03-10T00:53:36.656 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Mar 10 00:53:36 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-grafana-a[55281]: logger=migrator t=2024-03-10T00:53:36.625211866Z level=info msg="Executing migration" id="add index in alert_rule_version table on rule_org_id, rule_namespace_uid and rule_group columns" 2024-03-10T00:53:36.656 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Mar 10 00:53:36 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-grafana-a[55281]: logger=migrator t=2024-03-10T00:53:36.626194054Z level=info msg="Executing migration" id="alter alert_rule_version table data column to mediumtext in mysql" 2024-03-10T00:53:36.656 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Mar 10 00:53:36 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-grafana-a[55281]: logger=migrator t=2024-03-10T00:53:36.6265824Z level=info msg="Executing migration" id="add column for to alert_rule_version" 2024-03-10T00:53:36.656 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Mar 10 00:53:36 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-grafana-a[55281]: logger=migrator t=2024-03-10T00:53:36.627861164Z level=info msg="Executing migration" id="add column annotations to alert_rule_version" 2024-03-10T00:53:36.656 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Mar 10 00:53:36 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-grafana-a[55281]: logger=migrator t=2024-03-10T00:53:36.628820426Z level=info msg="Executing migration" id="add column labels to alert_rule_version" 2024-03-10T00:53:36.656 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Mar 10 00:53:36 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-grafana-a[55281]: logger=migrator t=2024-03-10T00:53:36.629638667Z level=info msg="Executing migration" id="add rule_group_idx column to alert_rule_version" 2024-03-10T00:53:36.656 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Mar 10 00:53:36 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-grafana-a[55281]: logger=migrator t=2024-03-10T00:53:36.63043842Z level=info msg="Executing migration" id="add is_paused column to alert_rule_versions table" 2024-03-10T00:53:36.656 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Mar 10 00:53:36 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-grafana-a[55281]: logger=migrator t=2024-03-10T00:53:36.63123162Z level=info msg="Executing migration" id="fix is_paused column for alert_rule_version table" 2024-03-10T00:53:36.656 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Mar 10 00:53:36 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-grafana-a[55281]: logger=migrator t=2024-03-10T00:53:36.631495044Z level=info msg="Executing migration" id=create_alert_configuration_table 2024-03-10T00:53:36.657 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Mar 10 00:53:36 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-grafana-a[55281]: logger=migrator t=2024-03-10T00:53:36.631846011Z level=info msg="Executing migration" id="Add column default in alert_configuration" 2024-03-10T00:53:36.967 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Mar 10 00:53:36 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-grafana-a[55281]: logger=migrator t=2024-03-10T00:53:36.632679004Z level=info msg="Executing migration" id="alert alert_configuration alertmanager_configuration column from TEXT to MEDIUMTEXT if mysql" 2024-03-10T00:53:36.968 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Mar 10 00:53:36 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-grafana-a[55281]: logger=migrator t=2024-03-10T00:53:36.632941872Z level=info msg="Executing migration" id="add column org_id in alert_configuration" 2024-03-10T00:53:36.968 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Mar 10 00:53:36 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-grafana-a[55281]: logger=migrator t=2024-03-10T00:53:36.633771325Z level=info msg="Executing migration" id="add index in alert_configuration table on org_id column" 2024-03-10T00:53:36.968 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Mar 10 00:53:36 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-grafana-a[55281]: logger=migrator t=2024-03-10T00:53:36.634168111Z level=info msg="Executing migration" id="add configuration_hash column to alert_configuration" 2024-03-10T00:53:36.968 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Mar 10 00:53:36 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-grafana-a[55281]: logger=migrator t=2024-03-10T00:53:36.634955419Z level=info msg="Executing migration" id=create_ngalert_configuration_table 2024-03-10T00:53:36.968 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Mar 10 00:53:36 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-grafana-a[55281]: logger=migrator t=2024-03-10T00:53:36.635326287Z level=info msg="Executing migration" id="add index in ngalert_configuration on org_id column" 2024-03-10T00:53:36.968 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Mar 10 00:53:36 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-grafana-a[55281]: logger=migrator t=2024-03-10T00:53:36.635711043Z level=info msg="Executing migration" id="add column send_alerts_to in ngalert_configuration" 2024-03-10T00:53:36.968 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Mar 10 00:53:36 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-grafana-a[55281]: logger=migrator t=2024-03-10T00:53:36.636530952Z level=info msg="Executing migration" id="create provenance_type table" 2024-03-10T00:53:36.968 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Mar 10 00:53:36 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-grafana-a[55281]: logger=migrator t=2024-03-10T00:53:36.63696471Z level=info msg="Executing migration" id="add index to uniquify (record_key, record_type, org_id) columns" 2024-03-10T00:53:36.968 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Mar 10 00:53:36 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-grafana-a[55281]: logger=migrator t=2024-03-10T00:53:36.637370359Z level=info msg="Executing migration" id="create alert_image table" 2024-03-10T00:53:36.968 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Mar 10 00:53:36 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-grafana-a[55281]: logger=migrator t=2024-03-10T00:53:36.637718513Z level=info msg="Executing migration" id="add unique index on token to alert_image table" 2024-03-10T00:53:36.968 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Mar 10 00:53:36 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-grafana-a[55281]: logger=migrator t=2024-03-10T00:53:36.638098222Z level=info msg="Executing migration" id="support longer URLs in alert_image table" 2024-03-10T00:53:36.969 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Mar 10 00:53:36 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-grafana-a[55281]: logger=migrator t=2024-03-10T00:53:36.638369502Z level=info msg="Executing migration" id=create_alert_configuration_history_table 2024-03-10T00:53:36.969 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Mar 10 00:53:36 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-grafana-a[55281]: logger=migrator t=2024-03-10T00:53:36.638732688Z level=info msg="Executing migration" id="drop non-unique orgID index on alert_configuration" 2024-03-10T00:53:36.969 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Mar 10 00:53:36 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-grafana-a[55281]: logger=migrator t=2024-03-10T00:53:36.639076Z level=info msg="Executing migration" id="drop unique orgID index on alert_configuration if exists" 2024-03-10T00:53:36.969 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Mar 10 00:53:36 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-grafana-a[55281]: logger=migrator t=2024-03-10T00:53:36.639157371Z level=warn msg="Skipping migration: Already executed, but not recorded in migration log" id="drop unique orgID index on alert_configuration if exists" 2024-03-10T00:53:36.969 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Mar 10 00:53:36 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-grafana-a[55281]: logger=migrator t=2024-03-10T00:53:36.639399189Z level=info msg="Executing migration" id="extract alertmanager configuration history to separate table" 2024-03-10T00:53:36.969 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Mar 10 00:53:36 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-grafana-a[55281]: logger=migrator t=2024-03-10T00:53:36.639686173Z level=info msg="Executing migration" id="add unique index on orgID to alert_configuration" 2024-03-10T00:53:36.969 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Mar 10 00:53:36 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-grafana-a[55281]: logger=migrator t=2024-03-10T00:53:36.64003093Z level=info msg="Executing migration" id="clear migration entry \"remove unified alerting data\"" 2024-03-10T00:53:36.969 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Mar 10 00:53:36 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-grafana-a[55281]: logger=migrator t=2024-03-10T00:53:36.640154939Z level=info msg="Executing migration" id="move dashboard alerts to unified alerting" 2024-03-10T00:53:36.969 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Mar 10 00:53:36 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-grafana-a[55281]: logger=migrator t=2024-03-10T00:53:36.640264205Z level=info msg="alerts found to migrate" alerts=0 2024-03-10T00:53:36.969 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Mar 10 00:53:36 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-grafana-a[55281]: logger=migrator t=2024-03-10T00:53:36.640613012Z level=info msg="Executing migration" id="create library_element table v1" 2024-03-10T00:53:36.969 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Mar 10 00:53:36 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-grafana-a[55281]: logger=migrator t=2024-03-10T00:53:36.64099867Z level=info msg="Executing migration" id="add index library_element org_id-folder_id-name-kind" 2024-03-10T00:53:36.969 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Mar 10 00:53:36 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-grafana-a[55281]: logger=migrator t=2024-03-10T00:53:36.641396051Z level=info msg="Executing migration" id="create library_element_connection table v1" 2024-03-10T00:53:36.970 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Mar 10 00:53:36 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-grafana-a[55281]: logger=migrator t=2024-03-10T00:53:36.641748034Z level=info msg="Executing migration" id="add index library_element_connection element_id-kind-connection_id" 2024-03-10T00:53:36.970 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Mar 10 00:53:36 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-grafana-a[55281]: logger=migrator t=2024-03-10T00:53:36.642144814Z level=info msg="Executing migration" id="add unique index library_element org_id_uid" 2024-03-10T00:53:36.970 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Mar 10 00:53:36 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-grafana-a[55281]: logger=migrator t=2024-03-10T00:53:36.642564741Z level=info msg="Executing migration" id="increase max description length to 2048" 2024-03-10T00:53:36.970 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Mar 10 00:53:36 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-grafana-a[55281]: logger=migrator t=2024-03-10T00:53:36.642792206Z level=info msg="Executing migration" id="clone move dashboard alerts to unified alerting" 2024-03-10T00:53:36.970 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Mar 10 00:53:36 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-grafana-a[55281]: logger=migrator t=2024-03-10T00:53:36.643089015Z level=info msg="Executing migration" id="create data_keys table" 2024-03-10T00:53:36.970 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Mar 10 00:53:36 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-grafana-a[55281]: logger=migrator t=2024-03-10T00:53:36.643474138Z level=info msg="Executing migration" id="create secrets table" 2024-03-10T00:53:36.970 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Mar 10 00:53:36 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-grafana-a[55281]: logger=migrator t=2024-03-10T00:53:36.643854884Z level=info msg="Executing migration" id="rename data_keys name column to id" 2024-03-10T00:53:36.970 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Mar 10 00:53:36 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-grafana-a[55281]: logger=migrator t=2024-03-10T00:53:36.648260344Z level=info msg="Executing migration" id="add name column into data_keys" 2024-03-10T00:53:36.970 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Mar 10 00:53:36 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-grafana-a[55281]: logger=migrator t=2024-03-10T00:53:36.649099021Z level=info msg="Executing migration" id="copy data_keys id column values into name" 2024-03-10T00:53:36.970 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Mar 10 00:53:36 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-grafana-a[55281]: logger=migrator t=2024-03-10T00:53:36.649388411Z level=info msg="Executing migration" id="rename data_keys name column to label" 2024-03-10T00:53:36.970 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Mar 10 00:53:36 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-grafana-a[55281]: logger=migrator t=2024-03-10T00:53:36.653593284Z level=info msg="Executing migration" id="rename data_keys id column back to name" 2024-03-10T00:53:36.970 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Mar 10 00:53:36 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-grafana-a[55281]: logger=migrator t=2024-03-10T00:53:36.65770462Z level=info msg="Executing migration" id="create kv_store table v1" 2024-03-10T00:53:36.971 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Mar 10 00:53:36 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-grafana-a[55281]: logger=migrator t=2024-03-10T00:53:36.658075972Z level=info msg="Executing migration" id="add index kv_store.org_id-namespace-key" 2024-03-10T00:53:36.972 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Mar 10 00:53:36 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-grafana-a[55281]: logger=migrator t=2024-03-10T00:53:36.658487698Z level=info msg="Executing migration" id="update dashboard_uid and panel_id from existing annotations" 2024-03-10T00:53:36.972 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Mar 10 00:53:36 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-grafana-a[55281]: logger=migrator t=2024-03-10T00:53:36.65877961Z level=info msg="Executing migration" id="create permission table" 2024-03-10T00:53:36.972 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Mar 10 00:53:36 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-grafana-a[55281]: logger=migrator t=2024-03-10T00:53:36.659135004Z level=info msg="Executing migration" id="add unique index permission.role_id" 2024-03-10T00:53:36.972 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Mar 10 00:53:36 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-grafana-a[55281]: logger=migrator t=2024-03-10T00:53:36.659537423Z level=info msg="Executing migration" id="add unique index role_id_action_scope" 2024-03-10T00:53:36.972 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Mar 10 00:53:36 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-grafana-a[55281]: logger=migrator t=2024-03-10T00:53:36.659927145Z level=info msg="Executing migration" id="create role table" 2024-03-10T00:53:36.972 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Mar 10 00:53:36 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-grafana-a[55281]: logger=migrator t=2024-03-10T00:53:36.660293807Z level=info msg="Executing migration" id="add column display_name" 2024-03-10T00:53:36.972 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Mar 10 00:53:36 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-grafana-a[55281]: logger=migrator t=2024-03-10T00:53:36.661177276Z level=info msg="Executing migration" id="add column group_name" 2024-03-10T00:53:36.972 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Mar 10 00:53:36 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-grafana-a[55281]: logger=migrator t=2024-03-10T00:53:36.662029043Z level=info msg="Executing migration" id="add index role.org_id" 2024-03-10T00:53:36.972 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Mar 10 00:53:36 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-grafana-a[55281]: logger=migrator t=2024-03-10T00:53:36.662424844Z level=info msg="Executing migration" id="add unique index role_org_id_name" 2024-03-10T00:53:36.972 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Mar 10 00:53:36 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-grafana-a[55281]: logger=migrator t=2024-03-10T00:53:36.662895372Z level=info msg="Executing migration" id="add index role_org_id_uid" 2024-03-10T00:53:36.972 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Mar 10 00:53:36 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-grafana-a[55281]: logger=migrator t=2024-03-10T00:53:36.663317781Z level=info msg="Executing migration" id="create team role table" 2024-03-10T00:53:36.972 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Mar 10 00:53:36 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-grafana-a[55281]: logger=migrator t=2024-03-10T00:53:36.663671868Z level=info msg="Executing migration" id="add index team_role.org_id" 2024-03-10T00:53:36.972 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Mar 10 00:53:36 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-grafana-a[55281]: logger=migrator t=2024-03-10T00:53:36.664065109Z level=info msg="Executing migration" id="add unique index team_role_org_id_team_id_role_id" 2024-03-10T00:53:36.972 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Mar 10 00:53:36 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-grafana-a[55281]: logger=migrator t=2024-03-10T00:53:36.664487649Z level=info msg="Executing migration" id="add index team_role.team_id" 2024-03-10T00:53:36.973 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Mar 10 00:53:36 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-grafana-a[55281]: logger=migrator t=2024-03-10T00:53:36.664876882Z level=info msg="Executing migration" id="create user role table" 2024-03-10T00:53:36.973 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Mar 10 00:53:36 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-grafana-a[55281]: logger=migrator t=2024-03-10T00:53:36.666181224Z level=info msg="Executing migration" id="add index user_role.org_id" 2024-03-10T00:53:36.973 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Mar 10 00:53:36 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-grafana-a[55281]: logger=migrator t=2024-03-10T00:53:36.666580224Z level=info msg="Executing migration" id="add unique index user_role_org_id_user_id_role_id" 2024-03-10T00:53:36.973 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Mar 10 00:53:36 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-grafana-a[55281]: logger=migrator t=2024-03-10T00:53:36.666994484Z level=info msg="Executing migration" id="add index user_role.user_id" 2024-03-10T00:53:36.973 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Mar 10 00:53:36 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-grafana-a[55281]: logger=migrator t=2024-03-10T00:53:36.667413207Z level=info msg="Executing migration" id="create builtin role table" 2024-03-10T00:53:36.973 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Mar 10 00:53:36 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-grafana-a[55281]: logger=migrator t=2024-03-10T00:53:36.667810955Z level=info msg="Executing migration" id="add index builtin_role.role_id" 2024-03-10T00:53:36.973 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Mar 10 00:53:36 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-grafana-a[55281]: logger=migrator t=2024-03-10T00:53:36.668206936Z level=info msg="Executing migration" id="add index builtin_role.name" 2024-03-10T00:53:36.973 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Mar 10 00:53:36 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-grafana-a[55281]: logger=migrator t=2024-03-10T00:53:36.668598889Z level=info msg="Executing migration" id="Add column org_id to builtin_role table" 2024-03-10T00:53:36.973 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Mar 10 00:53:36 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-grafana-a[55281]: logger=migrator t=2024-03-10T00:53:36.669516939Z level=info msg="Executing migration" id="add index builtin_role.org_id" 2024-03-10T00:53:36.973 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Mar 10 00:53:36 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-grafana-a[55281]: logger=migrator t=2024-03-10T00:53:36.669948677Z level=info msg="Executing migration" id="add unique index builtin_role_org_id_role_id_role" 2024-03-10T00:53:36.974 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Mar 10 00:53:36 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-grafana-a[55281]: logger=migrator t=2024-03-10T00:53:36.670381198Z level=info msg="Executing migration" id="Remove unique index role_org_id_uid" 2024-03-10T00:53:36.974 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Mar 10 00:53:36 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-grafana-a[55281]: logger=migrator t=2024-03-10T00:53:36.670760386Z level=info msg="Executing migration" id="add unique index role.uid" 2024-03-10T00:53:36.974 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Mar 10 00:53:36 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-grafana-a[55281]: logger=migrator t=2024-03-10T00:53:36.671121326Z level=info msg="Executing migration" id="create seed assignment table" 2024-03-10T00:53:36.974 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Mar 10 00:53:36 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-grafana-a[55281]: logger=migrator t=2024-03-10T00:53:36.671467824Z level=info msg="Executing migration" id="add unique index builtin_role_role_name" 2024-03-10T00:53:36.974 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Mar 10 00:53:36 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-grafana-a[55281]: logger=migrator t=2024-03-10T00:53:36.671855111Z level=info msg="Executing migration" id="add column hidden to role table" 2024-03-10T00:53:36.974 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Mar 10 00:53:36 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-grafana-a[55281]: logger=migrator t=2024-03-10T00:53:36.672769842Z level=info msg="Executing migration" id="create query_history table v1" 2024-03-10T00:53:36.974 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Mar 10 00:53:36 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-grafana-a[55281]: logger=migrator t=2024-03-10T00:53:36.673161621Z level=info msg="Executing migration" id="add index query_history.org_id-created_by-datasource_uid" 2024-03-10T00:53:36.974 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Mar 10 00:53:36 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-grafana-a[55281]: logger=migrator t=2024-03-10T00:53:36.673554962Z level=info msg="Executing migration" id="alter table query_history alter column created_by type to bigint" 2024-03-10T00:53:36.974 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Mar 10 00:53:36 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-grafana-a[55281]: logger=migrator t=2024-03-10T00:53:36.673778989Z level=info msg="Executing migration" id="teams permissions migration" 2024-03-10T00:53:36.974 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Mar 10 00:53:36 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-grafana-a[55281]: logger=migrator t=2024-03-10T00:53:36.674107267Z level=info msg="Executing migration" id="dashboard permissions" 2024-03-10T00:53:36.974 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Mar 10 00:53:36 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-grafana-a[55281]: logger=migrator t=2024-03-10T00:53:36.674548897Z level=info msg="Executing migration" id="dashboard permissions uid scopes" 2024-03-10T00:53:36.974 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Mar 10 00:53:36 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-grafana-a[55281]: logger=migrator t=2024-03-10T00:53:36.674842633Z level=info msg="Executing migration" id="drop managed folder create actions" 2024-03-10T00:53:36.975 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Mar 10 00:53:36 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-grafana-a[55281]: logger=migrator t=2024-03-10T00:53:36.675074442Z level=info msg="Executing migration" id="alerting notification permissions" 2024-03-10T00:53:36.975 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Mar 10 00:53:36 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-grafana-a[55281]: logger=migrator t=2024-03-10T00:53:36.675398513Z level=info msg="Executing migration" id="create query_history_star table v1" 2024-03-10T00:53:36.975 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Mar 10 00:53:36 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-grafana-a[55281]: logger=migrator t=2024-03-10T00:53:36.675743352Z level=info msg="Executing migration" id="add index query_history.user_id-query_uid" 2024-03-10T00:53:36.975 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Mar 10 00:53:36 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-grafana-a[55281]: logger=migrator t=2024-03-10T00:53:36.676149334Z level=info msg="Executing migration" id="add column org_id in query_history_star" 2024-03-10T00:53:36.975 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Mar 10 00:53:36 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-grafana-a[55281]: logger=migrator t=2024-03-10T00:53:36.677049419Z level=info msg="Executing migration" id="alter table query_history_star_mig column user_id type to bigint" 2024-03-10T00:53:36.975 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Mar 10 00:53:36 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-grafana-a[55281]: logger=migrator t=2024-03-10T00:53:36.677320214Z level=info msg="Executing migration" id="create correlation table v1" 2024-03-10T00:53:36.975 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Mar 10 00:53:36 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-grafana-a[55281]: logger=migrator t=2024-03-10T00:53:36.677681716Z level=info msg="Executing migration" id="add index correlations.uid" 2024-03-10T00:53:36.975 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Mar 10 00:53:36 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-grafana-a[55281]: logger=migrator t=2024-03-10T00:53:36.678092394Z level=info msg="Executing migration" id="add index correlations.source_uid" 2024-03-10T00:53:36.975 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Mar 10 00:53:36 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-grafana-a[55281]: logger=migrator t=2024-03-10T00:53:36.678512007Z level=info msg="Executing migration" id="add correlation config column" 2024-03-10T00:53:36.975 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Mar 10 00:53:36 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-grafana-a[55281]: logger=migrator t=2024-03-10T00:53:36.679437119Z level=info msg="Executing migration" id="create entity_events table" 2024-03-10T00:53:36.976 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Mar 10 00:53:36 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-grafana-a[55281]: logger=migrator t=2024-03-10T00:53:36.679817951Z level=info msg="Executing migration" id="create dashboard public config v1" 2024-03-10T00:53:36.976 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Mar 10 00:53:36 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-grafana-a[55281]: logger=migrator t=2024-03-10T00:53:36.680195102Z level=info msg="Executing migration" id="drop index UQE_dashboard_public_config_uid - v1" 2024-03-10T00:53:36.976 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Mar 10 00:53:36 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-grafana-a[55281]: logger=migrator t=2024-03-10T00:53:36.680278081Z level=warn msg="Skipping migration: Already executed, but not recorded in migration log" id="drop index UQE_dashboard_public_config_uid - v1" 2024-03-10T00:53:36.976 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Mar 10 00:53:36 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-grafana-a[55281]: logger=migrator t=2024-03-10T00:53:36.680529261Z level=info msg="Executing migration" id="drop index IDX_dashboard_public_config_org_id_dashboard_uid - v1" 2024-03-10T00:53:36.976 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Mar 10 00:53:36 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-grafana-a[55281]: logger=migrator t=2024-03-10T00:53:36.680603103Z level=warn msg="Skipping migration: Already executed, but not recorded in migration log" id="drop index IDX_dashboard_public_config_org_id_dashboard_uid - v1" 2024-03-10T00:53:36.976 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Mar 10 00:53:36 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-grafana-a[55281]: logger=migrator t=2024-03-10T00:53:36.680804823Z level=info msg="Executing migration" id="Drop old dashboard public config table" 2024-03-10T00:53:36.976 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Mar 10 00:53:36 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-grafana-a[55281]: logger=migrator t=2024-03-10T00:53:36.681121673Z level=info msg="Executing migration" id="recreate dashboard public config v1" 2024-03-10T00:53:36.976 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Mar 10 00:53:36 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-grafana-a[55281]: logger=migrator t=2024-03-10T00:53:36.681496193Z level=info msg="Executing migration" id="create index UQE_dashboard_public_config_uid - v1" 2024-03-10T00:53:36.976 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Mar 10 00:53:36 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-grafana-a[55281]: logger=migrator t=2024-03-10T00:53:36.681885365Z level=info msg="Executing migration" id="create index IDX_dashboard_public_config_org_id_dashboard_uid - v1" 2024-03-10T00:53:36.976 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Mar 10 00:53:36 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-grafana-a[55281]: logger=migrator t=2024-03-10T00:53:36.682302173Z level=info msg="Executing migration" id="drop index UQE_dashboard_public_config_uid - v2" 2024-03-10T00:53:36.976 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Mar 10 00:53:36 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-grafana-a[55281]: logger=migrator t=2024-03-10T00:53:36.682671465Z level=info msg="Executing migration" id="drop index IDX_dashboard_public_config_org_id_dashboard_uid - v2" 2024-03-10T00:53:36.977 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Mar 10 00:53:36 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-grafana-a[55281]: logger=migrator t=2024-03-10T00:53:36.683109789Z level=info msg="Executing migration" id="Drop public config table" 2024-03-10T00:53:36.977 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Mar 10 00:53:36 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-grafana-a[55281]: logger=migrator t=2024-03-10T00:53:36.683487715Z level=info msg="Executing migration" id="Recreate dashboard public config v2" 2024-03-10T00:53:36.977 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Mar 10 00:53:36 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-grafana-a[55281]: logger=migrator t=2024-03-10T00:53:36.683839833Z level=info msg="Executing migration" id="create index UQE_dashboard_public_config_uid - v2" 2024-03-10T00:53:36.977 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Mar 10 00:53:36 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-grafana-a[55281]: logger=migrator t=2024-03-10T00:53:36.684293643Z level=info msg="Executing migration" id="create index IDX_dashboard_public_config_org_id_dashboard_uid - v2" 2024-03-10T00:53:36.977 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Mar 10 00:53:36 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-grafana-a[55281]: logger=migrator t=2024-03-10T00:53:36.68491479Z level=info msg="Executing migration" id="create index UQE_dashboard_public_config_access_token - v2" 2024-03-10T00:53:36.977 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Mar 10 00:53:36 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-grafana-a[55281]: logger=migrator t=2024-03-10T00:53:36.685485534Z level=info msg="Executing migration" id="Rename table dashboard_public_config to dashboard_public - v2" 2024-03-10T00:53:36.977 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Mar 10 00:53:36 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-grafana-a[55281]: logger=migrator t=2024-03-10T00:53:36.68865661Z level=info msg="Executing migration" id="add annotations_enabled column" 2024-03-10T00:53:36.977 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Mar 10 00:53:36 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-grafana-a[55281]: logger=migrator t=2024-03-10T00:53:36.689648655Z level=info msg="Executing migration" id="add time_selection_enabled column" 2024-03-10T00:53:36.978 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Mar 10 00:53:36 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-grafana-a[55281]: logger=migrator t=2024-03-10T00:53:36.690665611Z level=info msg="Executing migration" id="delete orphaned public dashboards" 2024-03-10T00:53:36.978 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Mar 10 00:53:36 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-grafana-a[55281]: logger=migrator t=2024-03-10T00:53:36.690946486Z level=info msg="Executing migration" id="add share column" 2024-03-10T00:53:36.978 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Mar 10 00:53:36 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-grafana-a[55281]: logger=migrator t=2024-03-10T00:53:36.691914208Z level=info msg="Executing migration" id="create default alerting folders" 2024-03-10T00:53:36.978 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Mar 10 00:53:36 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-grafana-a[55281]: logger=migrator t=2024-03-10T00:53:36.692325682Z level=info msg="Executing migration" id="create file table" 2024-03-10T00:53:36.978 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Mar 10 00:53:36 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-grafana-a[55281]: logger=migrator t=2024-03-10T00:53:36.69271716Z level=info msg="Executing migration" id="file table idx: path natural pk" 2024-03-10T00:53:36.978 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Mar 10 00:53:36 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-grafana-a[55281]: logger=migrator t=2024-03-10T00:53:36.69312498Z level=info msg="Executing migration" id="file table idx: parent_folder_path_hash fast folder retrieval" 2024-03-10T00:53:36.978 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Mar 10 00:53:36 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-grafana-a[55281]: logger=migrator t=2024-03-10T00:53:36.693559664Z level=info msg="Executing migration" id="create file_meta table" 2024-03-10T00:53:36.978 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Mar 10 00:53:36 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-grafana-a[55281]: logger=migrator t=2024-03-10T00:53:36.693913166Z level=info msg="Executing migration" id="file table idx: path key" 2024-03-10T00:53:36.978 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Mar 10 00:53:36 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-grafana-a[55281]: logger=migrator t=2024-03-10T00:53:36.694322299Z level=info msg="Executing migration" id="set path collation in file table" 2024-03-10T00:53:36.978 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Mar 10 00:53:36 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-grafana-a[55281]: logger=migrator t=2024-03-10T00:53:36.694769143Z level=info msg="Executing migration" id="managed permissions migration" 2024-03-10T00:53:36.978 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Mar 10 00:53:36 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-grafana-a[55281]: logger=migrator t=2024-03-10T00:53:36.695072922Z level=info msg="Executing migration" id="managed folder permissions alert actions migration" 2024-03-10T00:53:36.978 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Mar 10 00:53:36 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-grafana-a[55281]: logger=migrator t=2024-03-10T00:53:36.695377931Z level=info msg="Executing migration" id="RBAC action name migrator" 2024-03-10T00:53:36.979 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Mar 10 00:53:36 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-grafana-a[55281]: logger=migrator t=2024-03-10T00:53:36.695768104Z level=info msg="Executing migration" id="Add UID column to playlist" 2024-03-10T00:53:36.979 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Mar 10 00:53:36 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-grafana-a[55281]: logger=migrator t=2024-03-10T00:53:36.69677116Z level=info msg="Executing migration" id="Update uid column values in playlist" 2024-03-10T00:53:36.979 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Mar 10 00:53:36 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-grafana-a[55281]: logger=migrator t=2024-03-10T00:53:36.697050207Z level=info msg="Executing migration" id="Add index for uid in playlist" 2024-03-10T00:53:36.979 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Mar 10 00:53:36 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-grafana-a[55281]: logger=migrator t=2024-03-10T00:53:36.697459364Z level=info msg="Executing migration" id="update group index for alert rules" 2024-03-10T00:53:36.979 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Mar 10 00:53:36 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-grafana-a[55281]: logger=migrator t=2024-03-10T00:53:36.697828159Z level=info msg="Executing migration" id="managed folder permissions alert actions repeated migration" 2024-03-10T00:53:36.979 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Mar 10 00:53:36 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-grafana-a[55281]: logger=migrator t=2024-03-10T00:53:36.698083043Z level=info msg="Executing migration" id="admin only folder/dashboard permission" 2024-03-10T00:53:36.979 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Mar 10 00:53:36 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-grafana-a[55281]: logger=migrator t=2024-03-10T00:53:36.698423223Z level=info msg="Executing migration" id="add action column to seed_assignment" 2024-03-10T00:53:36.979 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Mar 10 00:53:36 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-grafana-a[55281]: logger=migrator t=2024-03-10T00:53:36.699464027Z level=info msg="Executing migration" id="add scope column to seed_assignment" 2024-03-10T00:53:36.979 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Mar 10 00:53:36 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-grafana-a[55281]: logger=migrator t=2024-03-10T00:53:36.700532528Z level=info msg="Executing migration" id="remove unique index builtin_role_role_name before nullable update" 2024-03-10T00:53:36.979 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Mar 10 00:53:36 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-grafana-a[55281]: logger=migrator t=2024-03-10T00:53:36.70093122Z level=info msg="Executing migration" id="update seed_assignment role_name column to nullable" 2024-03-10T00:53:36.979 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Mar 10 00:53:36 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-grafana-a[55281]: logger=migrator t=2024-03-10T00:53:36.71137512Z level=info msg="Executing migration" id="add unique index builtin_role_name back" 2024-03-10T00:53:36.980 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Mar 10 00:53:36 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-grafana-a[55281]: logger=migrator t=2024-03-10T00:53:36.711780238Z level=info msg="Executing migration" id="add unique index builtin_role_action_scope" 2024-03-10T00:53:36.980 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Mar 10 00:53:36 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-grafana-a[55281]: logger=migrator t=2024-03-10T00:53:36.712173151Z level=info msg="Executing migration" id="add primary key to seed_assigment" 2024-03-10T00:53:36.980 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Mar 10 00:53:36 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-grafana-a[55281]: logger=migrator t=2024-03-10T00:53:36.715513486Z level=info msg="Executing migration" id="managed folder permissions alert actions repeated fixed migration" 2024-03-10T00:53:36.980 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Mar 10 00:53:36 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-grafana-a[55281]: logger=migrator t=2024-03-10T00:53:36.715797761Z level=info msg="Executing migration" id="migrate external alertmanagers to datsourcse" 2024-03-10T00:53:36.980 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Mar 10 00:53:36 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-grafana-a[55281]: logger=migrator t=2024-03-10T00:53:36.716077497Z level=info msg="Executing migration" id="create folder table" 2024-03-10T00:53:36.980 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Mar 10 00:53:36 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-grafana-a[55281]: logger=migrator t=2024-03-10T00:53:36.716464508Z level=info msg="Executing migration" id="Add index for parent_uid" 2024-03-10T00:53:36.980 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Mar 10 00:53:36 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-grafana-a[55281]: logger=migrator t=2024-03-10T00:53:36.71686359Z level=info msg="Executing migration" id="Add unique index for folder.uid and folder.org_id" 2024-03-10T00:53:36.980 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Mar 10 00:53:36 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-grafana-a[55281]: logger=migrator t=2024-03-10T00:53:36.717272283Z level=info msg="Executing migration" id="Update folder title length" 2024-03-10T00:53:36.980 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Mar 10 00:53:36 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-grafana-a[55281]: logger=migrator t=2024-03-10T00:53:36.717521277Z level=info msg="Executing migration" id="Add unique index for folder.title and folder.parent_uid" 2024-03-10T00:53:36.980 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Mar 10 00:53:36 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-grafana-a[55281]: logger=migrator t=2024-03-10T00:53:36.717898087Z level=info msg="migrations completed" performed=485 skipped=0 duration=256.0418ms 2024-03-10T00:53:36.980 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Mar 10 00:53:36 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-grafana-a[55281]: logger=sqlstore t=2024-03-10T00:53:36.718283379Z level=info msg="Created default organization" 2024-03-10T00:53:36.980 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Mar 10 00:53:36 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-grafana-a[55281]: logger=plugin.loader t=2024-03-10T00:53:36.775186867Z level=info msg="Plugin registered" pluginID=input 2024-03-10T00:53:36.980 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Mar 10 00:53:36 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-grafana-a[55281]: logger=plugin.loader t=2024-03-10T00:53:36.809862489Z level=info msg="Plugin registered" pluginID=grafana-piechart-panel 2024-03-10T00:53:36.981 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Mar 10 00:53:36 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-grafana-a[55281]: logger=plugin.loader t=2024-03-10T00:53:36.809896739Z level=info msg="Plugin registered" pluginID=vonage-status-panel 2024-03-10T00:53:36.981 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Mar 10 00:53:36 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-grafana-a[55281]: logger=secrets t=2024-03-10T00:53:36.809950895Z level=info msg="Envelope encryption state" enabled=true currentprovider=secretKey.v1 2024-03-10T00:53:36.981 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Mar 10 00:53:36 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-grafana-a[55281]: logger=query_data t=2024-03-10T00:53:36.812118443Z level=info msg="Query Service initialization" 2024-03-10T00:53:36.981 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Mar 10 00:53:36 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-grafana-a[55281]: logger=live.push_http t=2024-03-10T00:53:36.814530422Z level=info msg="Live Push Gateway initialization" 2024-03-10T00:53:37.274 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:53:36 smithi170 ceph-mon[27620]: pgmap v32: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-10T00:53:37.274 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:53:36 smithi170 ceph-mon[27620]: from='mgr.14634 172.21.15.106:0/2720500564' entity='mgr.y' 2024-03-10T00:53:37.274 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:53:36 smithi170 ceph-mon[27620]: from='mgr.14634 172.21.15.106:0/2720500564' entity='mgr.y' 2024-03-10T00:53:37.275 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:53:36 smithi170 ceph-mon[27620]: from='mgr.14634 172.21.15.106:0/2720500564' entity='mgr.y' 2024-03-10T00:53:37.275 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:53:36 smithi170 ceph-mon[27620]: from='mgr.14634 172.21.15.106:0/2720500564' entity='mgr.y' 2024-03-10T00:53:37.279 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:53:36 smithi106 ceph-mon[25742]: pgmap v32: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-10T00:53:37.279 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:53:36 smithi106 ceph-mon[25742]: from='mgr.14634 172.21.15.106:0/2720500564' entity='mgr.y' 2024-03-10T00:53:37.279 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:53:36 smithi106 ceph-mon[25742]: from='mgr.14634 172.21.15.106:0/2720500564' entity='mgr.y' 2024-03-10T00:53:37.279 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:53:36 smithi106 ceph-mon[25742]: from='mgr.14634 172.21.15.106:0/2720500564' entity='mgr.y' 2024-03-10T00:53:37.279 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:53:36 smithi106 ceph-mon[25742]: from='mgr.14634 172.21.15.106:0/2720500564' entity='mgr.y' 2024-03-10T00:53:37.280 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:53:37 smithi106 ceph-mon[33460]: pgmap v32: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-10T00:53:37.280 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:53:37 smithi106 ceph-mon[33460]: from='mgr.14634 172.21.15.106:0/2720500564' entity='mgr.y' 2024-03-10T00:53:37.280 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:53:37 smithi106 ceph-mon[33460]: from='mgr.14634 172.21.15.106:0/2720500564' entity='mgr.y' 2024-03-10T00:53:37.280 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:53:37 smithi106 ceph-mon[33460]: from='mgr.14634 172.21.15.106:0/2720500564' entity='mgr.y' 2024-03-10T00:53:37.280 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:53:37 smithi106 ceph-mon[33460]: from='mgr.14634 172.21.15.106:0/2720500564' entity='mgr.y' 2024-03-10T00:53:37.818 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Mar 10 00:53:37 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-grafana-a[55281]: logger=infra.usagestats.collector t=2024-03-10T00:53:37.558004639Z level=info msg="registering usage stat providers" usageStatsProvidersLen=2 2024-03-10T00:53:37.818 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Mar 10 00:53:37 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-grafana-a[55281]: logger=provisioning.datasources t=2024-03-10T00:53:37.559054094Z level=info msg="inserting datasource from configuration " name=Dashboard1 uid=P43CA22E17D0F9596 2024-03-10T00:53:37.819 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Mar 10 00:53:37 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-grafana-a[55281]: logger=provisioning.datasources t=2024-03-10T00:53:37.576745859Z level=info msg="inserting datasource from configuration " name=Loki uid=P8E80F9AEF21F6940 2024-03-10T00:53:37.819 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Mar 10 00:53:37 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-grafana-a[55281]: logger=provisioning.alerting t=2024-03-10T00:53:37.590582627Z level=info msg="starting to provision alerting" 2024-03-10T00:53:37.819 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Mar 10 00:53:37 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-grafana-a[55281]: logger=provisioning.alerting t=2024-03-10T00:53:37.590623009Z level=info msg="finished to provision alerting" 2024-03-10T00:53:37.819 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Mar 10 00:53:37 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-grafana-a[55281]: logger=grafanaStorageLogger t=2024-03-10T00:53:37.590832749Z level=info msg="storage starting" 2024-03-10T00:53:37.819 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Mar 10 00:53:37 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-grafana-a[55281]: logger=ngalert.state.manager t=2024-03-10T00:53:37.590879999Z level=info msg="Warming state cache for startup" 2024-03-10T00:53:37.819 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Mar 10 00:53:37 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-grafana-a[55281]: logger=http.server t=2024-03-10T00:53:37.59252765Z level=info msg="HTTP Server Listen" address=[::]:3000 protocol=https subUrl= socket= 2024-03-10T00:53:37.819 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Mar 10 00:53:37 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-grafana-a[55281]: logger=ngalert.state.manager t=2024-03-10T00:53:37.601672357Z level=info msg="State cache has been initialized" states=0 duration=10.791369ms 2024-03-10T00:53:37.819 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Mar 10 00:53:37 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-grafana-a[55281]: logger=ticker t=2024-03-10T00:53:37.601730319Z level=info msg=starting first_tick=2024-03-10T00:53:40Z 2024-03-10T00:53:37.819 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Mar 10 00:53:37 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-grafana-a[55281]: logger=ngalert.multiorg.alertmanager t=2024-03-10T00:53:37.601737872Z level=info msg="starting MultiOrg Alertmanager" 2024-03-10T00:53:37.819 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Mar 10 00:53:37 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-grafana-a[55281]: logger=sqlstore.transactions t=2024-03-10T00:53:37.617290315Z level=info msg="Database locked, sleeping then retrying" error="database is locked" retry=0 code="database is locked" 2024-03-10T00:53:38.178 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:53:38 smithi106 ceph-mon[25742]: Deploying daemon node-exporter.a on smithi106 2024-03-10T00:53:38.178 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:53:38 smithi106 ceph-mon[25742]: from='mgr.14634 172.21.15.106:0/2720500564' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.10", "id": [1, 2]}]: dispatch 2024-03-10T00:53:38.178 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:53:38 smithi106 ceph-mon[25742]: from='mgr.14634 172.21.15.106:0/2720500564' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.12", "id": [1, 5]}]: dispatch 2024-03-10T00:53:38.178 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:53:38 smithi106 ceph-mon[25742]: from='mgr.14634 172.21.15.106:0/2720500564' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1d", "id": [1, 2]}]: dispatch 2024-03-10T00:53:38.178 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:53:38 smithi106 ceph-mon[25742]: from='mgr.14634 172.21.15.106:0/2720500564' entity='mgr.y' 2024-03-10T00:53:38.178 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:53:38 smithi106 ceph-mon[25742]: from='mgr.14634 172.21.15.106:0/2720500564' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-10T00:53:38.179 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:53:38 smithi106 ceph-mon[33460]: Deploying daemon node-exporter.a on smithi106 2024-03-10T00:53:38.179 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:53:38 smithi106 ceph-mon[33460]: from='mgr.14634 172.21.15.106:0/2720500564' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.10", "id": [1, 2]}]: dispatch 2024-03-10T00:53:38.179 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:53:38 smithi106 ceph-mon[33460]: from='mgr.14634 172.21.15.106:0/2720500564' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.12", "id": [1, 5]}]: dispatch 2024-03-10T00:53:38.179 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:53:38 smithi106 ceph-mon[33460]: from='mgr.14634 172.21.15.106:0/2720500564' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1d", "id": [1, 2]}]: dispatch 2024-03-10T00:53:38.179 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:53:38 smithi106 ceph-mon[33460]: from='mgr.14634 172.21.15.106:0/2720500564' entity='mgr.y' 2024-03-10T00:53:38.179 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:53:38 smithi106 ceph-mon[33460]: from='mgr.14634 172.21.15.106:0/2720500564' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-10T00:53:38.210 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:53:38 smithi170 ceph-mon[27620]: Deploying daemon node-exporter.a on smithi106 2024-03-10T00:53:38.211 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:53:38 smithi170 ceph-mon[27620]: from='mgr.14634 172.21.15.106:0/2720500564' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.10", "id": [1, 2]}]: dispatch 2024-03-10T00:53:38.211 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:53:38 smithi170 ceph-mon[27620]: from='mgr.14634 172.21.15.106:0/2720500564' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.12", "id": [1, 5]}]: dispatch 2024-03-10T00:53:38.211 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:53:38 smithi170 ceph-mon[27620]: from='mgr.14634 172.21.15.106:0/2720500564' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1d", "id": [1, 2]}]: dispatch 2024-03-10T00:53:38.211 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:53:38 smithi170 ceph-mon[27620]: from='mgr.14634 172.21.15.106:0/2720500564' entity='mgr.y' 2024-03-10T00:53:38.211 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:53:38 smithi170 ceph-mon[27620]: from='mgr.14634 172.21.15.106:0/2720500564' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-10T00:53:38.774 INFO:journalctl@ceph.iscsi.iscsi.a.smithi170.stdout:Mar 10 00:53:38 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-iscsi-iscsi-a[50134]: debug there is no tcmu-runner data available 2024-03-10T00:53:39.020 INFO:teuthology.orchestra.run.smithi106.stdout:Backend: cephadm 2024-03-10T00:53:39.020 INFO:teuthology.orchestra.run.smithi106.stdout:Available: Yes 2024-03-10T00:53:39.021 INFO:teuthology.orchestra.run.smithi106.stdout:Paused: No 2024-03-10T00:53:39.049 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:53:39 smithi170 ceph-mon[27620]: pgmap v33: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-10T00:53:39.049 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:53:39 smithi170 ceph-mon[27620]: from='mgr.14634 172.21.15.106:0/2720500564' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.10", "id": [1, 2]}]': finished 2024-03-10T00:53:39.050 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:53:39 smithi170 ceph-mon[27620]: from='mgr.14634 172.21.15.106:0/2720500564' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.12", "id": [1, 5]}]': finished 2024-03-10T00:53:39.050 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:53:39 smithi170 ceph-mon[27620]: from='mgr.14634 172.21.15.106:0/2720500564' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1d", "id": [1, 2]}]': finished 2024-03-10T00:53:39.050 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:53:39 smithi170 ceph-mon[27620]: osdmap e70: 8 total, 8 up, 8 in 2024-03-10T00:53:39.338 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:53:39 smithi106 ceph-mon[25742]: pgmap v33: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-10T00:53:39.338 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:53:39 smithi106 ceph-mon[25742]: from='mgr.14634 172.21.15.106:0/2720500564' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.10", "id": [1, 2]}]': finished 2024-03-10T00:53:39.338 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:53:39 smithi106 ceph-mon[25742]: from='mgr.14634 172.21.15.106:0/2720500564' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.12", "id": [1, 5]}]': finished 2024-03-10T00:53:39.338 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:53:39 smithi106 ceph-mon[25742]: from='mgr.14634 172.21.15.106:0/2720500564' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1d", "id": [1, 2]}]': finished 2024-03-10T00:53:39.338 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:53:39 smithi106 ceph-mon[25742]: osdmap e70: 8 total, 8 up, 8 in 2024-03-10T00:53:39.338 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:53:39 smithi106 ceph-mon[33460]: pgmap v33: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-10T00:53:39.339 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:53:39 smithi106 ceph-mon[33460]: from='mgr.14634 172.21.15.106:0/2720500564' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.10", "id": [1, 2]}]': finished 2024-03-10T00:53:39.339 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:53:39 smithi106 ceph-mon[33460]: from='mgr.14634 172.21.15.106:0/2720500564' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.12", "id": [1, 5]}]': finished 2024-03-10T00:53:39.339 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:53:39 smithi106 ceph-mon[33460]: from='mgr.14634 172.21.15.106:0/2720500564' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1d", "id": [1, 2]}]': finished 2024-03-10T00:53:39.339 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:53:39 smithi106 ceph-mon[33460]: osdmap e70: 8 total, 8 up, 8 in 2024-03-10T00:53:40.003 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:831bc188fa69f143bbba69ac76119e784500e8e3 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 75c37498-de77-11ee-95c7-87774f69a715 -- bash -c 'ceph orch ps' 2024-03-10T00:53:40.203 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:53:40 smithi106 ceph-mon[33460]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-10T00:53:40.203 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:53:40 smithi106 ceph-mon[33460]: osdmap e71: 8 total, 8 up, 8 in 2024-03-10T00:53:40.203 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:53:40 smithi106 ceph-mon[33460]: from='client.24674 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-10T00:53:40.203 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:53:40 smithi106 ceph-mon[25742]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-10T00:53:40.203 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:53:40 smithi106 ceph-mon[25742]: osdmap e71: 8 total, 8 up, 8 in 2024-03-10T00:53:40.203 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:53:40 smithi106 ceph-mon[25742]: from='client.24674 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-10T00:53:40.268 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/75c37498-de77-11ee-95c7-87774f69a715/mon.a/config 2024-03-10T00:53:40.274 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:53:40 smithi170 ceph-mon[27620]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-10T00:53:40.274 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:53:40 smithi170 ceph-mon[27620]: osdmap e71: 8 total, 8 up, 8 in 2024-03-10T00:53:40.274 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:53:40 smithi170 ceph-mon[27620]: from='client.24674 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-10T00:53:41.274 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:53:41 smithi170 ceph-mon[27620]: pgmap v36: 132 pgs: 3 peering, 129 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-10T00:53:41.274 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:53:41 smithi170 ceph-mon[27620]: Health check failed: Reduced data availability: 3 pgs peering (PG_AVAILABILITY) 2024-03-10T00:53:41.338 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:53:41 smithi106 ceph-mon[25742]: pgmap v36: 132 pgs: 3 peering, 129 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-10T00:53:41.338 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:53:41 smithi106 ceph-mon[25742]: Health check failed: Reduced data availability: 3 pgs peering (PG_AVAILABILITY) 2024-03-10T00:53:41.339 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:53:41 smithi106 ceph-mon[33460]: pgmap v36: 132 pgs: 3 peering, 129 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-10T00:53:41.339 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:53:41 smithi106 ceph-mon[33460]: Health check failed: Reduced data availability: 3 pgs peering (PG_AVAILABILITY) 2024-03-10T00:53:41.794 INFO:journalctl@ceph.mgr.y.smithi106.stdout:Mar 10 00:53:41 smithi106 ceph-75c37498-de77-11ee-95c7-87774f69a715-mgr-y[25997]: ::ffff:172.21.15.170 - - [10/Mar/2024:00:53:41] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-10T00:53:42.522 INFO:teuthology.orchestra.run.smithi106.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-10T00:53:42.523 INFO:teuthology.orchestra.run.smithi106.stdout:agent.smithi106 smithi106 running 86s ago 6m - - 2024-03-10T00:53:42.523 INFO:teuthology.orchestra.run.smithi106.stdout:agent.smithi170 smithi170 running 88s ago 6m - - 2024-03-10T00:53:42.523 INFO:teuthology.orchestra.run.smithi106.stdout:alertmanager.a smithi106 *:9093,9094 starting - - - - 2024-03-10T00:53:42.523 INFO:teuthology.orchestra.run.smithi106.stdout:grafana.a smithi170 *:3000 starting - - - - 2024-03-10T00:53:42.523 INFO:teuthology.orchestra.run.smithi106.stdout:iscsi.iscsi.a smithi170 *:5000 running (98s) 88s ago 98s - - 3.6 01a4f6a6bb3e 8e6ee66d69a2 2024-03-10T00:53:42.523 INFO:teuthology.orchestra.run.smithi106.stdout:mgr.x smithi170 *:8443,8765 running (5m) 88s ago 5m - - 19.0.0-1875-g831bc188 01a4f6a6bb3e 5fd8a349b80c 2024-03-10T00:53:42.523 INFO:teuthology.orchestra.run.smithi106.stdout:mgr.y smithi106 *:9283,8765 running (7m) 86s ago 7m 512M - 19.0.0-1875-g831bc188 01a4f6a6bb3e 52b050b0e011 2024-03-10T00:53:42.523 INFO:teuthology.orchestra.run.smithi106.stdout:mon.a smithi106 running (7m) 86s ago 7m 44.9M 2048M 19.0.0-1875-g831bc188 01a4f6a6bb3e 97a03d7ed617 2024-03-10T00:53:42.523 INFO:teuthology.orchestra.run.smithi106.stdout:mon.b smithi170 running (6m) 88s ago 6m - 2048M 19.0.0-1875-g831bc188 01a4f6a6bb3e 4883b5f0e419 2024-03-10T00:53:42.523 INFO:teuthology.orchestra.run.smithi106.stdout:mon.c smithi106 running (5m) 86s ago 5m 37.3M 2048M 19.0.0-1875-g831bc188 01a4f6a6bb3e 2411e393082d 2024-03-10T00:53:42.523 INFO:teuthology.orchestra.run.smithi106.stdout:osd.0 smithi106 running (5m) 86s ago 5m 57.6M 4096M 19.0.0-1875-g831bc188 01a4f6a6bb3e 80d749660f4d 2024-03-10T00:53:42.523 INFO:teuthology.orchestra.run.smithi106.stdout:osd.1 smithi106 running (4m) 86s ago 4m 55.6M 4096M 19.0.0-1875-g831bc188 01a4f6a6bb3e 2a4d6d60877d 2024-03-10T00:53:42.524 INFO:teuthology.orchestra.run.smithi106.stdout:osd.2 smithi106 running (4m) 86s ago 4m 42.8M 4096M 19.0.0-1875-g831bc188 01a4f6a6bb3e 487a0b16efc7 2024-03-10T00:53:42.524 INFO:teuthology.orchestra.run.smithi106.stdout:osd.3 smithi106 running (3m) 86s ago 3m 50.9M 4096M 19.0.0-1875-g831bc188 01a4f6a6bb3e eb35f2f39ae8 2024-03-10T00:53:42.524 INFO:teuthology.orchestra.run.smithi106.stdout:osd.4 smithi170 running (3m) 88s ago 3m - 4096M 19.0.0-1875-g831bc188 01a4f6a6bb3e 895388359a4e 2024-03-10T00:53:42.524 INFO:teuthology.orchestra.run.smithi106.stdout:osd.5 smithi170 running (3m) 88s ago 3m - 4096M 19.0.0-1875-g831bc188 01a4f6a6bb3e 6993eeb86c16 2024-03-10T00:53:42.524 INFO:teuthology.orchestra.run.smithi106.stdout:osd.6 smithi170 running (2m) 88s ago 2m - 4096M 19.0.0-1875-g831bc188 01a4f6a6bb3e b3230350a17e 2024-03-10T00:53:42.524 INFO:teuthology.orchestra.run.smithi106.stdout:osd.7 smithi170 running (2m) 88s ago 2m - 4096M 19.0.0-1875-g831bc188 01a4f6a6bb3e 4a8bf4bdf3aa 2024-03-10T00:53:42.524 INFO:teuthology.orchestra.run.smithi106.stdout:prometheus.a smithi170 *:9095 starting - - - - 2024-03-10T00:53:42.524 INFO:teuthology.orchestra.run.smithi106.stdout:rgw.foo.a smithi106 *:80 running (111s) 86s ago 110s 75.3M - 19.0.0-1875-g831bc188 01a4f6a6bb3e d9af1c13f460 2024-03-10T00:53:43.274 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:53:43 smithi170 ceph-mon[27620]: pgmap v37: 132 pgs: 3 peering, 129 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-10T00:53:43.338 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:53:43 smithi106 ceph-mon[25742]: pgmap v37: 132 pgs: 3 peering, 129 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-10T00:53:43.338 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:53:43 smithi106 ceph-mon[33460]: pgmap v37: 132 pgs: 3 peering, 129 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-10T00:53:43.826 INFO:journalctl@ceph.node-exporter.a.smithi106.stdout:Mar 10 00:53:43 smithi106 bash[68165]: Writing manifest to image destination 2024-03-10T00:53:43.903 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:831bc188fa69f143bbba69ac76119e784500e8e3 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 75c37498-de77-11ee-95c7-87774f69a715 -- bash -c 'ceph orch ls' 2024-03-10T00:53:44.088 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:53:44 smithi106 ceph-mon[33460]: from='client.24680 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-10T00:53:44.088 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:53:44 smithi106 ceph-mon[25742]: from='client.24680 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-10T00:53:44.088 INFO:journalctl@ceph.node-exporter.a.smithi106.stdout:Mar 10 00:53:43 smithi106 podman[68165]: 2024-03-10 00:53:43.762639948 +0000 UTC m=+3.663282266 image pull 0da6a335fe1356545476b749c68f022c897de3a2139e8f0054f6937349ee2b83 quay.io/prometheus/node-exporter:v1.5.0 2024-03-10T00:53:44.089 INFO:journalctl@ceph.node-exporter.a.smithi106.stdout:Mar 10 00:53:44 smithi106 podman[68165]: 2024-03-10 00:53:44.011392455 +0000 UTC m=+3.912034762 container create fdb97a00c98ec0b681c0342dbd819bc2663f981caccbdb3dfd380c352759bdb7 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-75c37498-de77-11ee-95c7-87774f69a715-node-exporter-a, maintainer=The Prometheus Authors ) 2024-03-10T00:53:44.274 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:53:44 smithi170 ceph-mon[27620]: from='client.24680 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-10T00:53:44.399 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/75c37498-de77-11ee-95c7-87774f69a715/mon.a/config 2024-03-10T00:53:44.521 INFO:journalctl@ceph.node-exporter.a.smithi106.stdout:Mar 10 00:53:44 smithi106 podman[68165]: 2024-03-10 00:53:44.255410745 +0000 UTC m=+4.156053058 container init fdb97a00c98ec0b681c0342dbd819bc2663f981caccbdb3dfd380c352759bdb7 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-75c37498-de77-11ee-95c7-87774f69a715-node-exporter-a, maintainer=The Prometheus Authors ) 2024-03-10T00:53:44.522 INFO:journalctl@ceph.node-exporter.a.smithi106.stdout:Mar 10 00:53:44 smithi106 podman[68165]: 2024-03-10 00:53:44.260925525 +0000 UTC m=+4.161567823 container start fdb97a00c98ec0b681c0342dbd819bc2663f981caccbdb3dfd380c352759bdb7 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-75c37498-de77-11ee-95c7-87774f69a715-node-exporter-a, maintainer=The Prometheus Authors ) 2024-03-10T00:53:44.522 INFO:journalctl@ceph.node-exporter.a.smithi106.stdout:Mar 10 00:53:44 smithi106 ceph-75c37498-de77-11ee-95c7-87774f69a715-node-exporter-a[68449]: ts=2024-03-10T00:53:44.264Z caller=node_exporter.go:180 level=info msg="Starting node_exporter" version="(version=1.5.0, branch=HEAD, revision=1b48970ffcf5630534fb00bb0687d73c66d1c959)" 2024-03-10T00:53:44.522 INFO:journalctl@ceph.node-exporter.a.smithi106.stdout:Mar 10 00:53:44 smithi106 ceph-75c37498-de77-11ee-95c7-87774f69a715-node-exporter-a[68449]: ts=2024-03-10T00:53:44.264Z caller=node_exporter.go:181 level=info msg="Build context" build_context="(go=go1.19.3, user=root@6e7732a7b81b, date=20221129-18:59:09)" 2024-03-10T00:53:44.522 INFO:journalctl@ceph.node-exporter.a.smithi106.stdout:Mar 10 00:53:44 smithi106 ceph-75c37498-de77-11ee-95c7-87774f69a715-node-exporter-a[68449]: ts=2024-03-10T00:53:44.265Z caller=diskstats_common.go:111 level=info collector=diskstats msg="Parsed flag --collector.diskstats.device-exclude" flag=^(ram|loop|fd|(h|s|v|xv)d[a-z]|nvme\d+n\d+p)\d+$ 2024-03-10T00:53:44.522 INFO:journalctl@ceph.node-exporter.a.smithi106.stdout:Mar 10 00:53:44 smithi106 ceph-75c37498-de77-11ee-95c7-87774f69a715-node-exporter-a[68449]: ts=2024-03-10T00:53:44.265Z caller=diskstats_linux.go:264 level=error collector=diskstats msg="Failed to open directory, disabling udev device properties" path=/run/udev/data 2024-03-10T00:53:44.522 INFO:journalctl@ceph.node-exporter.a.smithi106.stdout:Mar 10 00:53:44 smithi106 ceph-75c37498-de77-11ee-95c7-87774f69a715-node-exporter-a[68449]: ts=2024-03-10T00:53:44.265Z caller=filesystem_common.go:111 level=info collector=filesystem msg="Parsed flag --collector.filesystem.mount-points-exclude" flag=^/(dev|proc|run/credentials/.+|sys|var/lib/docker/.+|var/lib/containers/storage/.+)($|/) 2024-03-10T00:53:44.522 INFO:journalctl@ceph.node-exporter.a.smithi106.stdout:Mar 10 00:53:44 smithi106 ceph-75c37498-de77-11ee-95c7-87774f69a715-node-exporter-a[68449]: ts=2024-03-10T00:53:44.265Z caller=filesystem_common.go:113 level=info collector=filesystem msg="Parsed flag --collector.filesystem.fs-types-exclude" flag=^(autofs|binfmt_misc|bpf|cgroup2?|configfs|debugfs|devpts|devtmpfs|fusectl|hugetlbfs|iso9660|mqueue|nsfs|overlay|proc|procfs|pstore|rpc_pipefs|securityfs|selinuxfs|squashfs|sysfs|tracefs)$ 2024-03-10T00:53:44.522 INFO:journalctl@ceph.node-exporter.a.smithi106.stdout:Mar 10 00:53:44 smithi106 ceph-75c37498-de77-11ee-95c7-87774f69a715-node-exporter-a[68449]: ts=2024-03-10T00:53:44.265Z caller=node_exporter.go:110 level=info msg="Enabled collectors" 2024-03-10T00:53:44.522 INFO:journalctl@ceph.node-exporter.a.smithi106.stdout:Mar 10 00:53:44 smithi106 ceph-75c37498-de77-11ee-95c7-87774f69a715-node-exporter-a[68449]: ts=2024-03-10T00:53:44.266Z caller=node_exporter.go:117 level=info collector=arp 2024-03-10T00:53:44.522 INFO:journalctl@ceph.node-exporter.a.smithi106.stdout:Mar 10 00:53:44 smithi106 ceph-75c37498-de77-11ee-95c7-87774f69a715-node-exporter-a[68449]: ts=2024-03-10T00:53:44.266Z caller=node_exporter.go:117 level=info collector=bcache 2024-03-10T00:53:44.522 INFO:journalctl@ceph.node-exporter.a.smithi106.stdout:Mar 10 00:53:44 smithi106 ceph-75c37498-de77-11ee-95c7-87774f69a715-node-exporter-a[68449]: ts=2024-03-10T00:53:44.266Z caller=node_exporter.go:117 level=info collector=bonding 2024-03-10T00:53:44.522 INFO:journalctl@ceph.node-exporter.a.smithi106.stdout:Mar 10 00:53:44 smithi106 ceph-75c37498-de77-11ee-95c7-87774f69a715-node-exporter-a[68449]: ts=2024-03-10T00:53:44.266Z caller=node_exporter.go:117 level=info collector=btrfs 2024-03-10T00:53:44.522 INFO:journalctl@ceph.node-exporter.a.smithi106.stdout:Mar 10 00:53:44 smithi106 ceph-75c37498-de77-11ee-95c7-87774f69a715-node-exporter-a[68449]: ts=2024-03-10T00:53:44.266Z caller=node_exporter.go:117 level=info collector=conntrack 2024-03-10T00:53:44.523 INFO:journalctl@ceph.node-exporter.a.smithi106.stdout:Mar 10 00:53:44 smithi106 ceph-75c37498-de77-11ee-95c7-87774f69a715-node-exporter-a[68449]: ts=2024-03-10T00:53:44.266Z caller=node_exporter.go:117 level=info collector=cpu 2024-03-10T00:53:44.523 INFO:journalctl@ceph.node-exporter.a.smithi106.stdout:Mar 10 00:53:44 smithi106 ceph-75c37498-de77-11ee-95c7-87774f69a715-node-exporter-a[68449]: ts=2024-03-10T00:53:44.266Z caller=node_exporter.go:117 level=info collector=cpufreq 2024-03-10T00:53:44.523 INFO:journalctl@ceph.node-exporter.a.smithi106.stdout:Mar 10 00:53:44 smithi106 ceph-75c37498-de77-11ee-95c7-87774f69a715-node-exporter-a[68449]: ts=2024-03-10T00:53:44.266Z caller=node_exporter.go:117 level=info collector=diskstats 2024-03-10T00:53:44.523 INFO:journalctl@ceph.node-exporter.a.smithi106.stdout:Mar 10 00:53:44 smithi106 ceph-75c37498-de77-11ee-95c7-87774f69a715-node-exporter-a[68449]: ts=2024-03-10T00:53:44.266Z caller=node_exporter.go:117 level=info collector=dmi 2024-03-10T00:53:44.523 INFO:journalctl@ceph.node-exporter.a.smithi106.stdout:Mar 10 00:53:44 smithi106 ceph-75c37498-de77-11ee-95c7-87774f69a715-node-exporter-a[68449]: ts=2024-03-10T00:53:44.266Z caller=node_exporter.go:117 level=info collector=edac 2024-03-10T00:53:44.523 INFO:journalctl@ceph.node-exporter.a.smithi106.stdout:Mar 10 00:53:44 smithi106 ceph-75c37498-de77-11ee-95c7-87774f69a715-node-exporter-a[68449]: ts=2024-03-10T00:53:44.266Z caller=node_exporter.go:117 level=info collector=entropy 2024-03-10T00:53:44.523 INFO:journalctl@ceph.node-exporter.a.smithi106.stdout:Mar 10 00:53:44 smithi106 ceph-75c37498-de77-11ee-95c7-87774f69a715-node-exporter-a[68449]: ts=2024-03-10T00:53:44.266Z caller=node_exporter.go:117 level=info collector=fibrechannel 2024-03-10T00:53:44.523 INFO:journalctl@ceph.node-exporter.a.smithi106.stdout:Mar 10 00:53:44 smithi106 ceph-75c37498-de77-11ee-95c7-87774f69a715-node-exporter-a[68449]: ts=2024-03-10T00:53:44.266Z caller=node_exporter.go:117 level=info collector=filefd 2024-03-10T00:53:44.523 INFO:journalctl@ceph.node-exporter.a.smithi106.stdout:Mar 10 00:53:44 smithi106 ceph-75c37498-de77-11ee-95c7-87774f69a715-node-exporter-a[68449]: ts=2024-03-10T00:53:44.266Z caller=node_exporter.go:117 level=info collector=filesystem 2024-03-10T00:53:44.523 INFO:journalctl@ceph.node-exporter.a.smithi106.stdout:Mar 10 00:53:44 smithi106 ceph-75c37498-de77-11ee-95c7-87774f69a715-node-exporter-a[68449]: ts=2024-03-10T00:53:44.266Z caller=node_exporter.go:117 level=info collector=hwmon 2024-03-10T00:53:44.523 INFO:journalctl@ceph.node-exporter.a.smithi106.stdout:Mar 10 00:53:44 smithi106 ceph-75c37498-de77-11ee-95c7-87774f69a715-node-exporter-a[68449]: ts=2024-03-10T00:53:44.266Z caller=node_exporter.go:117 level=info collector=infiniband 2024-03-10T00:53:44.523 INFO:journalctl@ceph.node-exporter.a.smithi106.stdout:Mar 10 00:53:44 smithi106 ceph-75c37498-de77-11ee-95c7-87774f69a715-node-exporter-a[68449]: ts=2024-03-10T00:53:44.266Z caller=node_exporter.go:117 level=info collector=ipvs 2024-03-10T00:53:44.523 INFO:journalctl@ceph.node-exporter.a.smithi106.stdout:Mar 10 00:53:44 smithi106 ceph-75c37498-de77-11ee-95c7-87774f69a715-node-exporter-a[68449]: ts=2024-03-10T00:53:44.266Z caller=node_exporter.go:117 level=info collector=loadavg 2024-03-10T00:53:44.523 INFO:journalctl@ceph.node-exporter.a.smithi106.stdout:Mar 10 00:53:44 smithi106 ceph-75c37498-de77-11ee-95c7-87774f69a715-node-exporter-a[68449]: ts=2024-03-10T00:53:44.266Z caller=node_exporter.go:117 level=info collector=mdadm 2024-03-10T00:53:44.524 INFO:journalctl@ceph.node-exporter.a.smithi106.stdout:Mar 10 00:53:44 smithi106 ceph-75c37498-de77-11ee-95c7-87774f69a715-node-exporter-a[68449]: ts=2024-03-10T00:53:44.266Z caller=node_exporter.go:117 level=info collector=meminfo 2024-03-10T00:53:44.524 INFO:journalctl@ceph.node-exporter.a.smithi106.stdout:Mar 10 00:53:44 smithi106 ceph-75c37498-de77-11ee-95c7-87774f69a715-node-exporter-a[68449]: ts=2024-03-10T00:53:44.266Z caller=node_exporter.go:117 level=info collector=netclass 2024-03-10T00:53:44.524 INFO:journalctl@ceph.node-exporter.a.smithi106.stdout:Mar 10 00:53:44 smithi106 ceph-75c37498-de77-11ee-95c7-87774f69a715-node-exporter-a[68449]: ts=2024-03-10T00:53:44.266Z caller=node_exporter.go:117 level=info collector=netdev 2024-03-10T00:53:44.524 INFO:journalctl@ceph.node-exporter.a.smithi106.stdout:Mar 10 00:53:44 smithi106 ceph-75c37498-de77-11ee-95c7-87774f69a715-node-exporter-a[68449]: ts=2024-03-10T00:53:44.266Z caller=node_exporter.go:117 level=info collector=netstat 2024-03-10T00:53:44.524 INFO:journalctl@ceph.node-exporter.a.smithi106.stdout:Mar 10 00:53:44 smithi106 ceph-75c37498-de77-11ee-95c7-87774f69a715-node-exporter-a[68449]: ts=2024-03-10T00:53:44.266Z caller=node_exporter.go:117 level=info collector=nfs 2024-03-10T00:53:44.524 INFO:journalctl@ceph.node-exporter.a.smithi106.stdout:Mar 10 00:53:44 smithi106 ceph-75c37498-de77-11ee-95c7-87774f69a715-node-exporter-a[68449]: ts=2024-03-10T00:53:44.266Z caller=node_exporter.go:117 level=info collector=nfsd 2024-03-10T00:53:44.524 INFO:journalctl@ceph.node-exporter.a.smithi106.stdout:Mar 10 00:53:44 smithi106 ceph-75c37498-de77-11ee-95c7-87774f69a715-node-exporter-a[68449]: ts=2024-03-10T00:53:44.266Z caller=node_exporter.go:117 level=info collector=nvme 2024-03-10T00:53:44.524 INFO:journalctl@ceph.node-exporter.a.smithi106.stdout:Mar 10 00:53:44 smithi106 ceph-75c37498-de77-11ee-95c7-87774f69a715-node-exporter-a[68449]: ts=2024-03-10T00:53:44.266Z caller=node_exporter.go:117 level=info collector=os 2024-03-10T00:53:44.524 INFO:journalctl@ceph.node-exporter.a.smithi106.stdout:Mar 10 00:53:44 smithi106 ceph-75c37498-de77-11ee-95c7-87774f69a715-node-exporter-a[68449]: ts=2024-03-10T00:53:44.266Z caller=node_exporter.go:117 level=info collector=powersupplyclass 2024-03-10T00:53:44.524 INFO:journalctl@ceph.node-exporter.a.smithi106.stdout:Mar 10 00:53:44 smithi106 ceph-75c37498-de77-11ee-95c7-87774f69a715-node-exporter-a[68449]: ts=2024-03-10T00:53:44.266Z caller=node_exporter.go:117 level=info collector=pressure 2024-03-10T00:53:44.524 INFO:journalctl@ceph.node-exporter.a.smithi106.stdout:Mar 10 00:53:44 smithi106 ceph-75c37498-de77-11ee-95c7-87774f69a715-node-exporter-a[68449]: ts=2024-03-10T00:53:44.266Z caller=node_exporter.go:117 level=info collector=rapl 2024-03-10T00:53:44.524 INFO:journalctl@ceph.node-exporter.a.smithi106.stdout:Mar 10 00:53:44 smithi106 ceph-75c37498-de77-11ee-95c7-87774f69a715-node-exporter-a[68449]: ts=2024-03-10T00:53:44.266Z caller=node_exporter.go:117 level=info collector=schedstat 2024-03-10T00:53:44.525 INFO:journalctl@ceph.node-exporter.a.smithi106.stdout:Mar 10 00:53:44 smithi106 ceph-75c37498-de77-11ee-95c7-87774f69a715-node-exporter-a[68449]: ts=2024-03-10T00:53:44.266Z caller=node_exporter.go:117 level=info collector=selinux 2024-03-10T00:53:44.525 INFO:journalctl@ceph.node-exporter.a.smithi106.stdout:Mar 10 00:53:44 smithi106 ceph-75c37498-de77-11ee-95c7-87774f69a715-node-exporter-a[68449]: ts=2024-03-10T00:53:44.266Z caller=node_exporter.go:117 level=info collector=sockstat 2024-03-10T00:53:44.525 INFO:journalctl@ceph.node-exporter.a.smithi106.stdout:Mar 10 00:53:44 smithi106 ceph-75c37498-de77-11ee-95c7-87774f69a715-node-exporter-a[68449]: ts=2024-03-10T00:53:44.266Z caller=node_exporter.go:117 level=info collector=softnet 2024-03-10T00:53:44.525 INFO:journalctl@ceph.node-exporter.a.smithi106.stdout:Mar 10 00:53:44 smithi106 ceph-75c37498-de77-11ee-95c7-87774f69a715-node-exporter-a[68449]: ts=2024-03-10T00:53:44.266Z caller=node_exporter.go:117 level=info collector=stat 2024-03-10T00:53:44.525 INFO:journalctl@ceph.node-exporter.a.smithi106.stdout:Mar 10 00:53:44 smithi106 ceph-75c37498-de77-11ee-95c7-87774f69a715-node-exporter-a[68449]: ts=2024-03-10T00:53:44.266Z caller=node_exporter.go:117 level=info collector=tapestats 2024-03-10T00:53:44.525 INFO:journalctl@ceph.node-exporter.a.smithi106.stdout:Mar 10 00:53:44 smithi106 ceph-75c37498-de77-11ee-95c7-87774f69a715-node-exporter-a[68449]: ts=2024-03-10T00:53:44.266Z caller=node_exporter.go:117 level=info collector=textfile 2024-03-10T00:53:44.525 INFO:journalctl@ceph.node-exporter.a.smithi106.stdout:Mar 10 00:53:44 smithi106 ceph-75c37498-de77-11ee-95c7-87774f69a715-node-exporter-a[68449]: ts=2024-03-10T00:53:44.266Z caller=node_exporter.go:117 level=info collector=thermal_zone 2024-03-10T00:53:44.525 INFO:journalctl@ceph.node-exporter.a.smithi106.stdout:Mar 10 00:53:44 smithi106 ceph-75c37498-de77-11ee-95c7-87774f69a715-node-exporter-a[68449]: ts=2024-03-10T00:53:44.266Z caller=node_exporter.go:117 level=info collector=time 2024-03-10T00:53:44.525 INFO:journalctl@ceph.node-exporter.a.smithi106.stdout:Mar 10 00:53:44 smithi106 ceph-75c37498-de77-11ee-95c7-87774f69a715-node-exporter-a[68449]: ts=2024-03-10T00:53:44.266Z caller=node_exporter.go:117 level=info collector=udp_queues 2024-03-10T00:53:44.525 INFO:journalctl@ceph.node-exporter.a.smithi106.stdout:Mar 10 00:53:44 smithi106 ceph-75c37498-de77-11ee-95c7-87774f69a715-node-exporter-a[68449]: ts=2024-03-10T00:53:44.266Z caller=node_exporter.go:117 level=info collector=uname 2024-03-10T00:53:44.525 INFO:journalctl@ceph.node-exporter.a.smithi106.stdout:Mar 10 00:53:44 smithi106 ceph-75c37498-de77-11ee-95c7-87774f69a715-node-exporter-a[68449]: ts=2024-03-10T00:53:44.266Z caller=node_exporter.go:117 level=info collector=vmstat 2024-03-10T00:53:44.525 INFO:journalctl@ceph.node-exporter.a.smithi106.stdout:Mar 10 00:53:44 smithi106 ceph-75c37498-de77-11ee-95c7-87774f69a715-node-exporter-a[68449]: ts=2024-03-10T00:53:44.266Z caller=node_exporter.go:117 level=info collector=xfs 2024-03-10T00:53:44.525 INFO:journalctl@ceph.node-exporter.a.smithi106.stdout:Mar 10 00:53:44 smithi106 ceph-75c37498-de77-11ee-95c7-87774f69a715-node-exporter-a[68449]: ts=2024-03-10T00:53:44.266Z caller=node_exporter.go:117 level=info collector=zfs 2024-03-10T00:53:44.525 INFO:journalctl@ceph.node-exporter.a.smithi106.stdout:Mar 10 00:53:44 smithi106 ceph-75c37498-de77-11ee-95c7-87774f69a715-node-exporter-a[68449]: ts=2024-03-10T00:53:44.266Z caller=tls_config.go:232 level=info msg="Listening on" address=[::]:9100 2024-03-10T00:53:44.526 INFO:journalctl@ceph.node-exporter.a.smithi106.stdout:Mar 10 00:53:44 smithi106 ceph-75c37498-de77-11ee-95c7-87774f69a715-node-exporter-a[68449]: ts=2024-03-10T00:53:44.266Z caller=tls_config.go:235 level=info msg="TLS is disabled." http2=false address=[::]:9100 2024-03-10T00:53:44.526 INFO:journalctl@ceph.node-exporter.a.smithi106.stdout:Mar 10 00:53:44 smithi106 bash[68165]: fdb97a00c98ec0b681c0342dbd819bc2663f981caccbdb3dfd380c352759bdb7 2024-03-10T00:53:44.526 INFO:journalctl@ceph.node-exporter.a.smithi106.stdout:Mar 10 00:53:44 smithi106 systemd[1]: Started Ceph node-exporter.a for 75c37498-de77-11ee-95c7-87774f69a715. 2024-03-10T00:53:45.088 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:53:45 smithi106 ceph-mon[25742]: pgmap v38: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 41 B/s, 1 objects/s recovering 2024-03-10T00:53:45.088 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:53:45 smithi106 ceph-mon[25742]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 3 pgs peering) 2024-03-10T00:53:45.088 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:53:45 smithi106 ceph-mon[25742]: Cluster is now healthy 2024-03-10T00:53:45.088 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:53:45 smithi106 ceph-mon[25742]: from='mgr.14634 172.21.15.106:0/2720500564' entity='mgr.y' 2024-03-10T00:53:45.088 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:53:45 smithi106 ceph-mon[25742]: from='mgr.14634 172.21.15.106:0/2720500564' entity='mgr.y' 2024-03-10T00:53:45.089 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:53:45 smithi106 ceph-mon[25742]: from='mgr.14634 172.21.15.106:0/2720500564' entity='mgr.y' 2024-03-10T00:53:45.089 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:53:45 smithi106 ceph-mon[33460]: pgmap v38: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 41 B/s, 1 objects/s recovering 2024-03-10T00:53:45.089 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:53:45 smithi106 ceph-mon[33460]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 3 pgs peering) 2024-03-10T00:53:45.089 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:53:45 smithi106 ceph-mon[33460]: Cluster is now healthy 2024-03-10T00:53:45.089 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:53:45 smithi106 ceph-mon[33460]: from='mgr.14634 172.21.15.106:0/2720500564' entity='mgr.y' 2024-03-10T00:53:45.089 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:53:45 smithi106 ceph-mon[33460]: from='mgr.14634 172.21.15.106:0/2720500564' entity='mgr.y' 2024-03-10T00:53:45.089 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:53:45 smithi106 ceph-mon[33460]: from='mgr.14634 172.21.15.106:0/2720500564' entity='mgr.y' 2024-03-10T00:53:45.368 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:53:45 smithi170 ceph-mon[27620]: pgmap v38: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 41 B/s, 1 objects/s recovering 2024-03-10T00:53:45.368 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:53:45 smithi170 ceph-mon[27620]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 3 pgs peering) 2024-03-10T00:53:45.368 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:53:45 smithi170 ceph-mon[27620]: Cluster is now healthy 2024-03-10T00:53:45.368 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:53:45 smithi170 ceph-mon[27620]: from='mgr.14634 172.21.15.106:0/2720500564' entity='mgr.y' 2024-03-10T00:53:45.369 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:53:45 smithi170 ceph-mon[27620]: from='mgr.14634 172.21.15.106:0/2720500564' entity='mgr.y' 2024-03-10T00:53:45.369 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:53:45 smithi170 ceph-mon[27620]: from='mgr.14634 172.21.15.106:0/2720500564' entity='mgr.y' 2024-03-10T00:53:45.674 INFO:journalctl@ceph.node-exporter.b.smithi170.stdout:Mar 10 00:53:45 smithi170 systemd[1]: Starting Ceph node-exporter.b for 75c37498-de77-11ee-95c7-87774f69a715... 2024-03-10T00:53:46.024 INFO:journalctl@ceph.node-exporter.b.smithi170.stdout:Mar 10 00:53:45 smithi170 bash[56473]: Trying to pull quay.io/prometheus/node-exporter:v1.5.0... 2024-03-10T00:53:46.088 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:53:46 smithi106 ceph-mon[25742]: Deploying daemon node-exporter.b on smithi170 2024-03-10T00:53:46.088 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:53:46 smithi106 ceph-mon[33460]: Deploying daemon node-exporter.b on smithi170 2024-03-10T00:53:46.524 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:53:46 smithi170 ceph-mon[27620]: Deploying daemon node-exporter.b on smithi170 2024-03-10T00:53:46.524 INFO:journalctl@ceph.node-exporter.b.smithi170.stdout:Mar 10 00:53:46 smithi170 bash[56473]: Getting image source signatures 2024-03-10T00:53:46.524 INFO:journalctl@ceph.node-exporter.b.smithi170.stdout:Mar 10 00:53:46 smithi170 bash[56473]: Copying blob sha256:c0e87333d3805a7b7e1c25783dd25f27da0fe70a8e262eee7ed190c8dc843fa7 2024-03-10T00:53:46.524 INFO:journalctl@ceph.node-exporter.b.smithi170.stdout:Mar 10 00:53:46 smithi170 bash[56473]: Copying blob sha256:5c12815fee558b157dee7f7509dedbaba0a8379098858a65ec869e1f1526ea0c 2024-03-10T00:53:46.524 INFO:journalctl@ceph.node-exporter.b.smithi170.stdout:Mar 10 00:53:46 smithi170 bash[56473]: Copying blob sha256:22b70bddd3acadc892fca4c2af4260629bfda5dfd11ebc106a93ce24e752b5ed 2024-03-10T00:53:46.632 INFO:teuthology.orchestra.run.smithi106.stdout:NAME PORTS RUNNING REFRESHED AGE PLACEMENT 2024-03-10T00:53:46.632 INFO:teuthology.orchestra.run.smithi106.stdout:agent 2/2 92s ago 69s * 2024-03-10T00:53:46.633 INFO:teuthology.orchestra.run.smithi106.stdout:alertmanager ?:9093,9094 0/1 - 93s smithi106=a;count:1 2024-03-10T00:53:46.633 INFO:teuthology.orchestra.run.smithi106.stdout:grafana ?:3000 0/1 - 90s smithi170=a;count:1 2024-03-10T00:53:46.633 INFO:teuthology.orchestra.run.smithi106.stdout:iscsi.datapool ?:5000 1/1 92s ago 106s smithi170=iscsi.a;count:1 2024-03-10T00:53:46.633 INFO:teuthology.orchestra.run.smithi106.stdout:mgr 2/2 92s ago 5m smithi106=y;smithi170=x;count:2 2024-03-10T00:53:46.633 INFO:teuthology.orchestra.run.smithi106.stdout:mon 3/3 92s ago 6m smithi106:172.21.15.106=a;smithi106:[v2:172.21.15.106:3301,v1:172.21.15.106:6790]=c;smithi170:172.21.15.170=b;count:3 2024-03-10T00:53:46.633 INFO:teuthology.orchestra.run.smithi106.stdout:node-exporter ?:9100 0/2 - 97s smithi106=a;smithi170=b;count:2 2024-03-10T00:53:46.633 INFO:teuthology.orchestra.run.smithi106.stdout:osd 8 92s ago - 2024-03-10T00:53:46.633 INFO:teuthology.orchestra.run.smithi106.stdout:prometheus ?:9095 0/1 - 102s smithi170=a;count:1 2024-03-10T00:53:46.633 INFO:teuthology.orchestra.run.smithi106.stdout:rgw.foo.a ?:80 1/1 90s ago 114s smithi106=foo.a;count:1 2024-03-10T00:53:47.338 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:53:47 smithi106 ceph-mon[25742]: pgmap v39: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 41 B/s, 1 objects/s recovering 2024-03-10T00:53:47.338 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:53:47 smithi106 ceph-mon[33460]: pgmap v39: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 41 B/s, 1 objects/s recovering 2024-03-10T00:53:47.390 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:53:47 smithi170 ceph-mon[27620]: pgmap v39: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 41 B/s, 1 objects/s recovering 2024-03-10T00:53:47.596 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:831bc188fa69f143bbba69ac76119e784500e8e3 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 75c37498-de77-11ee-95c7-87774f69a715 -- bash -c 'ceph orch host ls' 2024-03-10T00:53:47.774 INFO:journalctl@ceph.node-exporter.b.smithi170.stdout:Mar 10 00:53:47 smithi170 bash[56473]: Copying config sha256:0da6a335fe1356545476b749c68f022c897de3a2139e8f0054f6937349ee2b83 2024-03-10T00:53:47.774 INFO:journalctl@ceph.node-exporter.b.smithi170.stdout:Mar 10 00:53:47 smithi170 bash[56473]: Writing manifest to image destination 2024-03-10T00:53:47.837 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/75c37498-de77-11ee-95c7-87774f69a715/mon.a/config 2024-03-10T00:53:48.157 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:53:48 smithi170 ceph-mon[27620]: from='client.14796 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-10T00:53:48.157 INFO:journalctl@ceph.node-exporter.b.smithi170.stdout:Mar 10 00:53:47 smithi170 podman[56473]: 2024-03-10 00:53:47.807757108 +0000 UTC m=+2.222230879 image pull 0da6a335fe1356545476b749c68f022c897de3a2139e8f0054f6937349ee2b83 quay.io/prometheus/node-exporter:v1.5.0 2024-03-10T00:53:48.158 INFO:journalctl@ceph.node-exporter.b.smithi170.stdout:Mar 10 00:53:47 smithi170 podman[56473]: 2024-03-10 00:53:47.906147298 +0000 UTC m=+2.320621064 container create 5b7019120d1fa43fb65545e472c525a122a52c3005e11fb93201070a4cb57cbf (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-75c37498-de77-11ee-95c7-87774f69a715-node-exporter-b, maintainer=The Prometheus Authors ) 2024-03-10T00:53:48.214 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:53:48 smithi106 ceph-mon[25742]: from='client.14796 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-10T00:53:48.214 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:53:48 smithi106 ceph-mon[33460]: from='client.14796 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-10T00:53:48.525 INFO:journalctl@ceph.node-exporter.b.smithi170.stdout:Mar 10 00:53:48 smithi170 podman[56473]: 2024-03-10 00:53:48.160442723 +0000 UTC m=+2.574916493 container init 5b7019120d1fa43fb65545e472c525a122a52c3005e11fb93201070a4cb57cbf (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-75c37498-de77-11ee-95c7-87774f69a715-node-exporter-b, maintainer=The Prometheus Authors ) 2024-03-10T00:53:48.525 INFO:journalctl@ceph.node-exporter.b.smithi170.stdout:Mar 10 00:53:48 smithi170 podman[56473]: 2024-03-10 00:53:48.166284599 +0000 UTC m=+2.580758366 container start 5b7019120d1fa43fb65545e472c525a122a52c3005e11fb93201070a4cb57cbf (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-75c37498-de77-11ee-95c7-87774f69a715-node-exporter-b, maintainer=The Prometheus Authors ) 2024-03-10T00:53:48.525 INFO:journalctl@ceph.node-exporter.b.smithi170.stdout:Mar 10 00:53:48 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-node-exporter-b[56524]: ts=2024-03-10T00:53:48.170Z caller=node_exporter.go:180 level=info msg="Starting node_exporter" version="(version=1.5.0, branch=HEAD, revision=1b48970ffcf5630534fb00bb0687d73c66d1c959)" 2024-03-10T00:53:48.525 INFO:journalctl@ceph.node-exporter.b.smithi170.stdout:Mar 10 00:53:48 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-node-exporter-b[56524]: ts=2024-03-10T00:53:48.170Z caller=node_exporter.go:181 level=info msg="Build context" build_context="(go=go1.19.3, user=root@6e7732a7b81b, date=20221129-18:59:09)" 2024-03-10T00:53:48.525 INFO:journalctl@ceph.node-exporter.b.smithi170.stdout:Mar 10 00:53:48 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-node-exporter-b[56524]: ts=2024-03-10T00:53:48.171Z caller=filesystem_common.go:111 level=info collector=filesystem msg="Parsed flag --collector.filesystem.mount-points-exclude" flag=^/(dev|proc|run/credentials/.+|sys|var/lib/docker/.+|var/lib/containers/storage/.+)($|/) 2024-03-10T00:53:48.525 INFO:journalctl@ceph.node-exporter.b.smithi170.stdout:Mar 10 00:53:48 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-node-exporter-b[56524]: ts=2024-03-10T00:53:48.171Z caller=filesystem_common.go:113 level=info collector=filesystem msg="Parsed flag --collector.filesystem.fs-types-exclude" flag=^(autofs|binfmt_misc|bpf|cgroup2?|configfs|debugfs|devpts|devtmpfs|fusectl|hugetlbfs|iso9660|mqueue|nsfs|overlay|proc|procfs|pstore|rpc_pipefs|securityfs|selinuxfs|squashfs|sysfs|tracefs)$ 2024-03-10T00:53:48.525 INFO:journalctl@ceph.node-exporter.b.smithi170.stdout:Mar 10 00:53:48 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-node-exporter-b[56524]: ts=2024-03-10T00:53:48.171Z caller=diskstats_common.go:111 level=info collector=diskstats msg="Parsed flag --collector.diskstats.device-exclude" flag=^(ram|loop|fd|(h|s|v|xv)d[a-z]|nvme\d+n\d+p)\d+$ 2024-03-10T00:53:48.525 INFO:journalctl@ceph.node-exporter.b.smithi170.stdout:Mar 10 00:53:48 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-node-exporter-b[56524]: ts=2024-03-10T00:53:48.171Z caller=diskstats_linux.go:264 level=error collector=diskstats msg="Failed to open directory, disabling udev device properties" path=/run/udev/data 2024-03-10T00:53:48.525 INFO:journalctl@ceph.node-exporter.b.smithi170.stdout:Mar 10 00:53:48 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-node-exporter-b[56524]: ts=2024-03-10T00:53:48.172Z caller=node_exporter.go:110 level=info msg="Enabled collectors" 2024-03-10T00:53:48.525 INFO:journalctl@ceph.node-exporter.b.smithi170.stdout:Mar 10 00:53:48 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-node-exporter-b[56524]: ts=2024-03-10T00:53:48.172Z caller=node_exporter.go:117 level=info collector=arp 2024-03-10T00:53:48.526 INFO:journalctl@ceph.node-exporter.b.smithi170.stdout:Mar 10 00:53:48 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-node-exporter-b[56524]: ts=2024-03-10T00:53:48.172Z caller=node_exporter.go:117 level=info collector=bcache 2024-03-10T00:53:48.526 INFO:journalctl@ceph.node-exporter.b.smithi170.stdout:Mar 10 00:53:48 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-node-exporter-b[56524]: ts=2024-03-10T00:53:48.172Z caller=node_exporter.go:117 level=info collector=bonding 2024-03-10T00:53:48.526 INFO:journalctl@ceph.node-exporter.b.smithi170.stdout:Mar 10 00:53:48 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-node-exporter-b[56524]: ts=2024-03-10T00:53:48.172Z caller=node_exporter.go:117 level=info collector=btrfs 2024-03-10T00:53:48.526 INFO:journalctl@ceph.node-exporter.b.smithi170.stdout:Mar 10 00:53:48 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-node-exporter-b[56524]: ts=2024-03-10T00:53:48.172Z caller=node_exporter.go:117 level=info collector=conntrack 2024-03-10T00:53:48.526 INFO:journalctl@ceph.node-exporter.b.smithi170.stdout:Mar 10 00:53:48 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-node-exporter-b[56524]: ts=2024-03-10T00:53:48.172Z caller=node_exporter.go:117 level=info collector=cpu 2024-03-10T00:53:48.526 INFO:journalctl@ceph.node-exporter.b.smithi170.stdout:Mar 10 00:53:48 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-node-exporter-b[56524]: ts=2024-03-10T00:53:48.172Z caller=node_exporter.go:117 level=info collector=cpufreq 2024-03-10T00:53:48.527 INFO:journalctl@ceph.node-exporter.b.smithi170.stdout:Mar 10 00:53:48 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-node-exporter-b[56524]: ts=2024-03-10T00:53:48.172Z caller=node_exporter.go:117 level=info collector=diskstats 2024-03-10T00:53:48.527 INFO:journalctl@ceph.node-exporter.b.smithi170.stdout:Mar 10 00:53:48 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-node-exporter-b[56524]: ts=2024-03-10T00:53:48.172Z caller=node_exporter.go:117 level=info collector=dmi 2024-03-10T00:53:48.527 INFO:journalctl@ceph.node-exporter.b.smithi170.stdout:Mar 10 00:53:48 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-node-exporter-b[56524]: ts=2024-03-10T00:53:48.172Z caller=node_exporter.go:117 level=info collector=edac 2024-03-10T00:53:48.527 INFO:journalctl@ceph.node-exporter.b.smithi170.stdout:Mar 10 00:53:48 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-node-exporter-b[56524]: ts=2024-03-10T00:53:48.172Z caller=node_exporter.go:117 level=info collector=entropy 2024-03-10T00:53:48.527 INFO:journalctl@ceph.node-exporter.b.smithi170.stdout:Mar 10 00:53:48 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-node-exporter-b[56524]: ts=2024-03-10T00:53:48.172Z caller=node_exporter.go:117 level=info collector=fibrechannel 2024-03-10T00:53:48.527 INFO:journalctl@ceph.node-exporter.b.smithi170.stdout:Mar 10 00:53:48 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-node-exporter-b[56524]: ts=2024-03-10T00:53:48.172Z caller=node_exporter.go:117 level=info collector=filefd 2024-03-10T00:53:48.527 INFO:journalctl@ceph.node-exporter.b.smithi170.stdout:Mar 10 00:53:48 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-node-exporter-b[56524]: ts=2024-03-10T00:53:48.172Z caller=node_exporter.go:117 level=info collector=filesystem 2024-03-10T00:53:48.527 INFO:journalctl@ceph.node-exporter.b.smithi170.stdout:Mar 10 00:53:48 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-node-exporter-b[56524]: ts=2024-03-10T00:53:48.172Z caller=node_exporter.go:117 level=info collector=hwmon 2024-03-10T00:53:48.527 INFO:journalctl@ceph.node-exporter.b.smithi170.stdout:Mar 10 00:53:48 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-node-exporter-b[56524]: ts=2024-03-10T00:53:48.172Z caller=node_exporter.go:117 level=info collector=infiniband 2024-03-10T00:53:48.527 INFO:journalctl@ceph.node-exporter.b.smithi170.stdout:Mar 10 00:53:48 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-node-exporter-b[56524]: ts=2024-03-10T00:53:48.172Z caller=node_exporter.go:117 level=info collector=ipvs 2024-03-10T00:53:48.527 INFO:journalctl@ceph.node-exporter.b.smithi170.stdout:Mar 10 00:53:48 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-node-exporter-b[56524]: ts=2024-03-10T00:53:48.172Z caller=node_exporter.go:117 level=info collector=loadavg 2024-03-10T00:53:48.527 INFO:journalctl@ceph.node-exporter.b.smithi170.stdout:Mar 10 00:53:48 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-node-exporter-b[56524]: ts=2024-03-10T00:53:48.172Z caller=node_exporter.go:117 level=info collector=mdadm 2024-03-10T00:53:48.527 INFO:journalctl@ceph.node-exporter.b.smithi170.stdout:Mar 10 00:53:48 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-node-exporter-b[56524]: ts=2024-03-10T00:53:48.172Z caller=node_exporter.go:117 level=info collector=meminfo 2024-03-10T00:53:48.528 INFO:journalctl@ceph.node-exporter.b.smithi170.stdout:Mar 10 00:53:48 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-node-exporter-b[56524]: ts=2024-03-10T00:53:48.172Z caller=node_exporter.go:117 level=info collector=netclass 2024-03-10T00:53:48.528 INFO:journalctl@ceph.node-exporter.b.smithi170.stdout:Mar 10 00:53:48 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-node-exporter-b[56524]: ts=2024-03-10T00:53:48.172Z caller=node_exporter.go:117 level=info collector=netdev 2024-03-10T00:53:48.528 INFO:journalctl@ceph.node-exporter.b.smithi170.stdout:Mar 10 00:53:48 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-node-exporter-b[56524]: ts=2024-03-10T00:53:48.172Z caller=node_exporter.go:117 level=info collector=netstat 2024-03-10T00:53:48.528 INFO:journalctl@ceph.node-exporter.b.smithi170.stdout:Mar 10 00:53:48 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-node-exporter-b[56524]: ts=2024-03-10T00:53:48.172Z caller=node_exporter.go:117 level=info collector=nfs 2024-03-10T00:53:48.528 INFO:journalctl@ceph.node-exporter.b.smithi170.stdout:Mar 10 00:53:48 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-node-exporter-b[56524]: ts=2024-03-10T00:53:48.172Z caller=node_exporter.go:117 level=info collector=nfsd 2024-03-10T00:53:48.528 INFO:journalctl@ceph.node-exporter.b.smithi170.stdout:Mar 10 00:53:48 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-node-exporter-b[56524]: ts=2024-03-10T00:53:48.172Z caller=node_exporter.go:117 level=info collector=nvme 2024-03-10T00:53:48.528 INFO:journalctl@ceph.node-exporter.b.smithi170.stdout:Mar 10 00:53:48 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-node-exporter-b[56524]: ts=2024-03-10T00:53:48.172Z caller=node_exporter.go:117 level=info collector=os 2024-03-10T00:53:48.528 INFO:journalctl@ceph.node-exporter.b.smithi170.stdout:Mar 10 00:53:48 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-node-exporter-b[56524]: ts=2024-03-10T00:53:48.172Z caller=node_exporter.go:117 level=info collector=powersupplyclass 2024-03-10T00:53:48.528 INFO:journalctl@ceph.node-exporter.b.smithi170.stdout:Mar 10 00:53:48 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-node-exporter-b[56524]: ts=2024-03-10T00:53:48.172Z caller=node_exporter.go:117 level=info collector=pressure 2024-03-10T00:53:48.528 INFO:journalctl@ceph.node-exporter.b.smithi170.stdout:Mar 10 00:53:48 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-node-exporter-b[56524]: ts=2024-03-10T00:53:48.172Z caller=node_exporter.go:117 level=info collector=rapl 2024-03-10T00:53:48.528 INFO:journalctl@ceph.node-exporter.b.smithi170.stdout:Mar 10 00:53:48 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-node-exporter-b[56524]: ts=2024-03-10T00:53:48.172Z caller=node_exporter.go:117 level=info collector=schedstat 2024-03-10T00:53:48.528 INFO:journalctl@ceph.node-exporter.b.smithi170.stdout:Mar 10 00:53:48 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-node-exporter-b[56524]: ts=2024-03-10T00:53:48.172Z caller=node_exporter.go:117 level=info collector=selinux 2024-03-10T00:53:48.528 INFO:journalctl@ceph.node-exporter.b.smithi170.stdout:Mar 10 00:53:48 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-node-exporter-b[56524]: ts=2024-03-10T00:53:48.172Z caller=node_exporter.go:117 level=info collector=sockstat 2024-03-10T00:53:48.528 INFO:journalctl@ceph.node-exporter.b.smithi170.stdout:Mar 10 00:53:48 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-node-exporter-b[56524]: ts=2024-03-10T00:53:48.172Z caller=node_exporter.go:117 level=info collector=softnet 2024-03-10T00:53:48.528 INFO:journalctl@ceph.node-exporter.b.smithi170.stdout:Mar 10 00:53:48 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-node-exporter-b[56524]: ts=2024-03-10T00:53:48.172Z caller=node_exporter.go:117 level=info collector=stat 2024-03-10T00:53:48.529 INFO:journalctl@ceph.node-exporter.b.smithi170.stdout:Mar 10 00:53:48 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-node-exporter-b[56524]: ts=2024-03-10T00:53:48.172Z caller=node_exporter.go:117 level=info collector=tapestats 2024-03-10T00:53:48.529 INFO:journalctl@ceph.node-exporter.b.smithi170.stdout:Mar 10 00:53:48 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-node-exporter-b[56524]: ts=2024-03-10T00:53:48.172Z caller=node_exporter.go:117 level=info collector=textfile 2024-03-10T00:53:48.529 INFO:journalctl@ceph.node-exporter.b.smithi170.stdout:Mar 10 00:53:48 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-node-exporter-b[56524]: ts=2024-03-10T00:53:48.172Z caller=node_exporter.go:117 level=info collector=thermal_zone 2024-03-10T00:53:48.529 INFO:journalctl@ceph.node-exporter.b.smithi170.stdout:Mar 10 00:53:48 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-node-exporter-b[56524]: ts=2024-03-10T00:53:48.172Z caller=node_exporter.go:117 level=info collector=time 2024-03-10T00:53:48.529 INFO:journalctl@ceph.node-exporter.b.smithi170.stdout:Mar 10 00:53:48 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-node-exporter-b[56524]: ts=2024-03-10T00:53:48.172Z caller=node_exporter.go:117 level=info collector=udp_queues 2024-03-10T00:53:48.529 INFO:journalctl@ceph.node-exporter.b.smithi170.stdout:Mar 10 00:53:48 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-node-exporter-b[56524]: ts=2024-03-10T00:53:48.172Z caller=node_exporter.go:117 level=info collector=uname 2024-03-10T00:53:48.529 INFO:journalctl@ceph.node-exporter.b.smithi170.stdout:Mar 10 00:53:48 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-node-exporter-b[56524]: ts=2024-03-10T00:53:48.172Z caller=node_exporter.go:117 level=info collector=vmstat 2024-03-10T00:53:48.529 INFO:journalctl@ceph.node-exporter.b.smithi170.stdout:Mar 10 00:53:48 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-node-exporter-b[56524]: ts=2024-03-10T00:53:48.172Z caller=node_exporter.go:117 level=info collector=xfs 2024-03-10T00:53:48.529 INFO:journalctl@ceph.node-exporter.b.smithi170.stdout:Mar 10 00:53:48 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-node-exporter-b[56524]: ts=2024-03-10T00:53:48.172Z caller=node_exporter.go:117 level=info collector=zfs 2024-03-10T00:53:48.529 INFO:journalctl@ceph.node-exporter.b.smithi170.stdout:Mar 10 00:53:48 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-node-exporter-b[56524]: ts=2024-03-10T00:53:48.172Z caller=tls_config.go:232 level=info msg="Listening on" address=[::]:9100 2024-03-10T00:53:48.529 INFO:journalctl@ceph.node-exporter.b.smithi170.stdout:Mar 10 00:53:48 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-node-exporter-b[56524]: ts=2024-03-10T00:53:48.172Z caller=tls_config.go:235 level=info msg="TLS is disabled." http2=false address=[::]:9100 2024-03-10T00:53:48.529 INFO:journalctl@ceph.node-exporter.b.smithi170.stdout:Mar 10 00:53:48 smithi170 bash[56473]: 5b7019120d1fa43fb65545e472c525a122a52c3005e11fb93201070a4cb57cbf 2024-03-10T00:53:48.529 INFO:journalctl@ceph.node-exporter.b.smithi170.stdout:Mar 10 00:53:48 smithi170 systemd[1]: Started Ceph node-exporter.b for 75c37498-de77-11ee-95c7-87774f69a715. 2024-03-10T00:53:49.024 INFO:journalctl@ceph.iscsi.iscsi.a.smithi170.stdout:Mar 10 00:53:48 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-iscsi-iscsi-a[50134]: debug there is no tcmu-runner data available 2024-03-10T00:53:49.338 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:53:49 smithi106 ceph-mon[25742]: pgmap v40: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s; 34 B/s, 1 objects/s recovering 2024-03-10T00:53:49.338 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:53:49 smithi106 ceph-mon[25742]: from='mgr.14634 172.21.15.106:0/2720500564' entity='mgr.y' 2024-03-10T00:53:49.338 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:53:49 smithi106 ceph-mon[25742]: from='mgr.14634 172.21.15.106:0/2720500564' entity='mgr.y' 2024-03-10T00:53:49.338 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:53:49 smithi106 ceph-mon[25742]: from='mgr.14634 172.21.15.106:0/2720500564' entity='mgr.y' 2024-03-10T00:53:49.338 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:53:49 smithi106 ceph-mon[25742]: from='mgr.14634 172.21.15.106:0/2720500564' entity='mgr.y' 2024-03-10T00:53:49.338 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:53:49 smithi106 ceph-mon[25742]: from='mgr.14634 172.21.15.106:0/2720500564' entity='mgr.y' 2024-03-10T00:53:49.338 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:53:49 smithi106 ceph-mon[25742]: from='mgr.14634 172.21.15.106:0/2720500564' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-10T00:53:49.338 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:53:49 smithi106 ceph-mon[25742]: from='mgr.14634 172.21.15.106:0/2720500564' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.agent.smithi106", "caps": []}]: dispatch 2024-03-10T00:53:49.339 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:53:49 smithi106 ceph-mon[25742]: from='mgr.14634 172.21.15.106:0/2720500564' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.agent.smithi170", "caps": []}]: dispatch 2024-03-10T00:53:49.339 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:53:49 smithi106 ceph-mon[33460]: pgmap v40: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s; 34 B/s, 1 objects/s recovering 2024-03-10T00:53:49.339 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:53:49 smithi106 ceph-mon[33460]: from='mgr.14634 172.21.15.106:0/2720500564' entity='mgr.y' 2024-03-10T00:53:49.339 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:53:49 smithi106 ceph-mon[33460]: from='mgr.14634 172.21.15.106:0/2720500564' entity='mgr.y' 2024-03-10T00:53:49.339 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:53:49 smithi106 ceph-mon[33460]: from='mgr.14634 172.21.15.106:0/2720500564' entity='mgr.y' 2024-03-10T00:53:49.339 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:53:49 smithi106 ceph-mon[33460]: from='mgr.14634 172.21.15.106:0/2720500564' entity='mgr.y' 2024-03-10T00:53:49.339 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:53:49 smithi106 ceph-mon[33460]: from='mgr.14634 172.21.15.106:0/2720500564' entity='mgr.y' 2024-03-10T00:53:49.339 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:53:49 smithi106 ceph-mon[33460]: from='mgr.14634 172.21.15.106:0/2720500564' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-10T00:53:49.339 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:53:49 smithi106 ceph-mon[33460]: from='mgr.14634 172.21.15.106:0/2720500564' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.agent.smithi106", "caps": []}]: dispatch 2024-03-10T00:53:49.339 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:53:49 smithi106 ceph-mon[33460]: from='mgr.14634 172.21.15.106:0/2720500564' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.agent.smithi170", "caps": []}]: dispatch 2024-03-10T00:53:49.369 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:53:49 smithi170 ceph-mon[27620]: pgmap v40: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s; 34 B/s, 1 objects/s recovering 2024-03-10T00:53:49.369 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:53:49 smithi170 ceph-mon[27620]: from='mgr.14634 172.21.15.106:0/2720500564' entity='mgr.y' 2024-03-10T00:53:49.369 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:53:49 smithi170 ceph-mon[27620]: from='mgr.14634 172.21.15.106:0/2720500564' entity='mgr.y' 2024-03-10T00:53:49.369 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:53:49 smithi170 ceph-mon[27620]: from='mgr.14634 172.21.15.106:0/2720500564' entity='mgr.y' 2024-03-10T00:53:49.369 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:53:49 smithi170 ceph-mon[27620]: from='mgr.14634 172.21.15.106:0/2720500564' entity='mgr.y' 2024-03-10T00:53:49.369 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:53:49 smithi170 ceph-mon[27620]: from='mgr.14634 172.21.15.106:0/2720500564' entity='mgr.y' 2024-03-10T00:53:49.369 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:53:49 smithi170 ceph-mon[27620]: from='mgr.14634 172.21.15.106:0/2720500564' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-10T00:53:49.370 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:53:49 smithi170 ceph-mon[27620]: from='mgr.14634 172.21.15.106:0/2720500564' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.agent.smithi106", "caps": []}]: dispatch 2024-03-10T00:53:49.370 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:53:49 smithi170 ceph-mon[27620]: from='mgr.14634 172.21.15.106:0/2720500564' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.agent.smithi170", "caps": []}]: dispatch 2024-03-10T00:53:50.063 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:53:50 smithi106 ceph-mon[33460]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-10T00:53:50.063 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:53:50 smithi106 ceph-mon[33460]: Reconfiguring daemon agent.smithi170 on smithi170 2024-03-10T00:53:50.063 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:53:50 smithi106 ceph-mon[33460]: from='mgr.14634 172.21.15.106:0/2720500564' entity='mgr.y' 2024-03-10T00:53:50.063 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:53:50 smithi106 ceph-mon[25742]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-10T00:53:50.063 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:53:50 smithi106 ceph-mon[25742]: Reconfiguring daemon agent.smithi170 on smithi170 2024-03-10T00:53:50.063 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:53:50 smithi106 ceph-mon[25742]: from='mgr.14634 172.21.15.106:0/2720500564' entity='mgr.y' 2024-03-10T00:53:50.262 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:53:50 smithi170 ceph-mon[27620]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-10T00:53:50.262 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:53:50 smithi170 ceph-mon[27620]: Reconfiguring daemon agent.smithi170 on smithi170 2024-03-10T00:53:50.262 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:53:50 smithi170 ceph-mon[27620]: from='mgr.14634 172.21.15.106:0/2720500564' entity='mgr.y' 2024-03-10T00:53:50.563 INFO:teuthology.orchestra.run.smithi106.stdout:HOST ADDR LABELS STATUS 2024-03-10T00:53:50.563 INFO:teuthology.orchestra.run.smithi106.stdout:smithi106 172.21.15.106 2024-03-10T00:53:50.563 INFO:teuthology.orchestra.run.smithi106.stdout:smithi170 172.21.15.170 2024-03-10T00:53:50.563 INFO:teuthology.orchestra.run.smithi106.stdout:2 hosts in cluster 2024-03-10T00:53:51.359 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:831bc188fa69f143bbba69ac76119e784500e8e3 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 75c37498-de77-11ee-95c7-87774f69a715 -- bash -c 'ceph orch device ls' 2024-03-10T00:53:51.385 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:53:51 smithi106 ceph-mon[25742]: Reconfiguring daemon agent.smithi106 on smithi106 2024-03-10T00:53:51.385 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:53:51 smithi106 ceph-mon[25742]: pgmap v41: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 968 B/s rd, 0 op/s; 31 B/s, 1 objects/s recovering 2024-03-10T00:53:51.385 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:53:51 smithi106 ceph-mon[25742]: from='mgr.14634 172.21.15.106:0/2720500564' entity='mgr.y' 2024-03-10T00:53:51.385 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:53:51 smithi106 ceph-mon[33460]: Reconfiguring daemon agent.smithi106 on smithi106 2024-03-10T00:53:51.386 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:53:51 smithi106 ceph-mon[33460]: pgmap v41: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 968 B/s rd, 0 op/s; 31 B/s, 1 objects/s recovering 2024-03-10T00:53:51.386 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:53:51 smithi106 ceph-mon[33460]: from='mgr.14634 172.21.15.106:0/2720500564' entity='mgr.y' 2024-03-10T00:53:51.524 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:53:51 smithi170 ceph-mon[27620]: Reconfiguring daemon agent.smithi106 on smithi106 2024-03-10T00:53:51.524 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:53:51 smithi170 ceph-mon[27620]: pgmap v41: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 968 B/s rd, 0 op/s; 31 B/s, 1 objects/s recovering 2024-03-10T00:53:51.524 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:53:51 smithi170 ceph-mon[27620]: from='mgr.14634 172.21.15.106:0/2720500564' entity='mgr.y' 2024-03-10T00:53:51.766 INFO:journalctl@ceph.mgr.y.smithi106.stdout:Mar 10 00:53:51 smithi106 ceph-75c37498-de77-11ee-95c7-87774f69a715-mgr-y[25997]: ::ffff:172.21.15.170 - - [10/Mar/2024:00:53:51] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-10T00:53:52.224 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/75c37498-de77-11ee-95c7-87774f69a715/mon.a/config 2024-03-10T00:53:52.373 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:53:52 smithi170 ceph-mon[27620]: from='client.24692 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-10T00:53:52.444 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:53:52 smithi106 ceph-mon[25742]: from='client.24692 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-10T00:53:52.444 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:53:52 smithi106 ceph-mon[33460]: from='client.24692 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-10T00:53:53.524 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:53:53 smithi170 ceph-mon[27620]: pgmap v42: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 27 B/s, 0 objects/s recovering 2024-03-10T00:53:53.524 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:53:53 smithi170 ceph-mon[27620]: from='mgr.14634 172.21.15.106:0/2720500564' entity='mgr.y' 2024-03-10T00:53:53.524 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:53:53 smithi170 ceph-mon[27620]: from='mgr.14634 172.21.15.106:0/2720500564' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-10T00:53:53.559 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:53:53 smithi106 ceph-mon[25742]: pgmap v42: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 27 B/s, 0 objects/s recovering 2024-03-10T00:53:53.559 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:53:53 smithi106 ceph-mon[25742]: from='mgr.14634 172.21.15.106:0/2720500564' entity='mgr.y' 2024-03-10T00:53:53.559 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:53:53 smithi106 ceph-mon[25742]: from='mgr.14634 172.21.15.106:0/2720500564' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-10T00:53:53.559 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:53:53 smithi106 ceph-mon[33460]: pgmap v42: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 27 B/s, 0 objects/s recovering 2024-03-10T00:53:53.559 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:53:53 smithi106 ceph-mon[33460]: from='mgr.14634 172.21.15.106:0/2720500564' entity='mgr.y' 2024-03-10T00:53:53.559 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:53:53 smithi106 ceph-mon[33460]: from='mgr.14634 172.21.15.106:0/2720500564' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-10T00:53:55.274 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:53:55 smithi170 ceph-mon[27620]: pgmap v43: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 27 B/s, 0 objects/s recovering 2024-03-10T00:53:55.292 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:53:55 smithi106 ceph-mon[33460]: pgmap v43: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 27 B/s, 0 objects/s recovering 2024-03-10T00:53:55.292 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:53:55 smithi106 ceph-mon[25742]: pgmap v43: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 27 B/s, 0 objects/s recovering 2024-03-10T00:53:57.338 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:53:57 smithi106 ceph-mon[25742]: pgmap v44: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-10T00:53:57.338 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:53:57 smithi106 ceph-mon[33460]: pgmap v44: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-10T00:53:57.350 INFO:teuthology.orchestra.run.smithi106.stdout:HOST PATH TYPE DEVICE ID SIZE AVAILABLE REFRESHED REJECT REASONS 2024-03-10T00:53:57.350 INFO:teuthology.orchestra.run.smithi106.stdout:smithi106 /dev/nvme0n1 ssd INTEL_SSDPEDMD400G4_PHFT6204013R400BGN 372G No 2m ago Has a FileSystem, LVM detected 2024-03-10T00:53:57.350 INFO:teuthology.orchestra.run.smithi106.stdout:smithi106 /dev/nvme1n1 ssd Linux_360593095f039aa499ca 89.4G No 2m ago Has a FileSystem, Insufficient space (<10 extents) on vgs, LVM detected 2024-03-10T00:53:57.350 INFO:teuthology.orchestra.run.smithi106.stdout:smithi106 /dev/nvme2n1 ssd Linux_2c097f3a9f2b69440f66 89.4G No 2m ago Has a FileSystem, Insufficient space (<10 extents) on vgs, LVM detected 2024-03-10T00:53:57.350 INFO:teuthology.orchestra.run.smithi106.stdout:smithi106 /dev/nvme3n1 ssd Linux_43400f8a9a7d42abdb0c 89.4G No 2m ago Has a FileSystem, Insufficient space (<10 extents) on vgs, LVM detected 2024-03-10T00:53:57.350 INFO:teuthology.orchestra.run.smithi106.stdout:smithi106 /dev/nvme4n1 ssd Linux_907b47e64be117f399fb 89.4G No 2m ago Has a FileSystem, Insufficient space (<10 extents) on vgs, LVM detected 2024-03-10T00:53:57.350 INFO:teuthology.orchestra.run.smithi106.stdout:smithi106 /dev/sda hdd ATA_ST1000NM0033-9ZM173_Z1W5QHPJ 931G No 2m ago Has partitions 2024-03-10T00:53:57.350 INFO:teuthology.orchestra.run.smithi106.stdout:smithi170 /dev/nvme0n1 ssd INTEL_SSDPED1D480GA_PHMB7476002R480DGN 447G No 103s ago Has a FileSystem, LVM detected 2024-03-10T00:53:57.350 INFO:teuthology.orchestra.run.smithi106.stdout:smithi170 /dev/nvme1n1 ssd Linux_0bec78b1e11737015da3 89.4G No 103s ago Has a FileSystem, Insufficient space (<10 extents) on vgs, LVM detected 2024-03-10T00:53:57.351 INFO:teuthology.orchestra.run.smithi106.stdout:smithi170 /dev/nvme2n1 ssd Linux_7f67e114c7bb103ae901 89.4G No 103s ago Has a FileSystem, Insufficient space (<10 extents) on vgs, LVM detected 2024-03-10T00:53:57.351 INFO:teuthology.orchestra.run.smithi106.stdout:smithi170 /dev/nvme3n1 ssd Linux_e5e9296aa0892d5aa3a2 89.4G No 103s ago Has a FileSystem, Insufficient space (<10 extents) on vgs, LVM detected 2024-03-10T00:53:57.351 INFO:teuthology.orchestra.run.smithi106.stdout:smithi170 /dev/nvme4n1 ssd Linux_23d280e791f50e30f40c 89.4G No 103s ago Has a FileSystem, Insufficient space (<10 extents) on vgs, LVM detected 2024-03-10T00:53:57.351 INFO:teuthology.orchestra.run.smithi106.stdout:smithi170 /dev/sda hdd ATA_ST1000NM0033-9ZM173_Z1W5XK1Y 931G No 103s ago Has partitions 2024-03-10T00:53:57.524 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:53:57 smithi170 ceph-mon[27620]: pgmap v44: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-10T00:53:57.986 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:831bc188fa69f143bbba69ac76119e784500e8e3 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 75c37498-de77-11ee-95c7-87774f69a715 -- bash -c 'ceph orch ls --format yaml' 2024-03-10T00:53:59.004 INFO:journalctl@ceph.iscsi.iscsi.a.smithi170.stdout:Mar 10 00:53:58 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-iscsi-iscsi-a[50134]: debug there is no tcmu-runner data available 2024-03-10T00:53:59.274 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:53:59 smithi170 ceph-mon[27620]: from='client.14808 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-10T00:53:59.274 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:53:59 smithi170 ceph-mon[27620]: pgmap v45: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-10T00:53:59.338 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:53:59 smithi106 ceph-mon[25742]: from='client.14808 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-10T00:53:59.338 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:53:59 smithi106 ceph-mon[25742]: pgmap v45: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-10T00:53:59.338 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:53:59 smithi106 ceph-mon[33460]: from='client.14808 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-10T00:53:59.338 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:53:59 smithi106 ceph-mon[33460]: pgmap v45: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-10T00:53:59.415 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/75c37498-de77-11ee-95c7-87774f69a715/mon.a/config 2024-03-10T00:54:00.338 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:54:00 smithi106 ceph-mon[25742]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-10T00:54:00.338 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:54:00 smithi106 ceph-mon[25742]: from='mgr.14634 172.21.15.106:0/2720500564' entity='mgr.y' 2024-03-10T00:54:00.338 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:54:00 smithi106 ceph-mon[25742]: from='mgr.14634 172.21.15.106:0/2720500564' entity='mgr.y' 2024-03-10T00:54:00.339 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:54:00 smithi106 ceph-mon[33460]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-10T00:54:00.339 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:54:00 smithi106 ceph-mon[33460]: from='mgr.14634 172.21.15.106:0/2720500564' entity='mgr.y' 2024-03-10T00:54:00.339 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:54:00 smithi106 ceph-mon[33460]: from='mgr.14634 172.21.15.106:0/2720500564' entity='mgr.y' 2024-03-10T00:54:00.431 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:54:00 smithi170 ceph-mon[27620]: from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-10T00:54:00.431 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:54:00 smithi170 ceph-mon[27620]: from='mgr.14634 172.21.15.106:0/2720500564' entity='mgr.y' 2024-03-10T00:54:00.432 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:54:00 smithi170 ceph-mon[27620]: from='mgr.14634 172.21.15.106:0/2720500564' entity='mgr.y' 2024-03-10T00:54:01.338 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:54:01 smithi106 ceph-mon[25742]: pgmap v46: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-10T00:54:01.338 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:54:01 smithi106 ceph-mon[33460]: pgmap v46: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-10T00:54:01.524 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:54:01 smithi170 ceph-mon[27620]: pgmap v46: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-10T00:54:01.701 INFO:journalctl@ceph.mgr.y.smithi106.stdout:Mar 10 00:54:01 smithi106 ceph-75c37498-de77-11ee-95c7-87774f69a715-mgr-y[25997]: ::ffff:172.21.15.170 - - [10/Mar/2024:00:54:01] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-10T00:54:02.338 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:54:02 smithi106 ceph-mon[33460]: from='mgr.14634 172.21.15.106:0/2720500564' entity='mgr.y' 2024-03-10T00:54:02.338 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:54:02 smithi106 ceph-mon[33460]: from='mgr.14634 172.21.15.106:0/2720500564' entity='mgr.y' 2024-03-10T00:54:02.338 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:54:02 smithi106 ceph-mon[33460]: pgmap v47: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-10T00:54:02.338 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:54:02 smithi106 ceph-mon[33460]: from='mgr.14634 172.21.15.106:0/2720500564' entity='mgr.y' 2024-03-10T00:54:02.338 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:54:02 smithi106 ceph-mon[33460]: from='mgr.14634 172.21.15.106:0/2720500564' entity='mgr.y' 2024-03-10T00:54:02.339 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:54:02 smithi106 ceph-mon[25742]: from='mgr.14634 172.21.15.106:0/2720500564' entity='mgr.y' 2024-03-10T00:54:02.339 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:54:02 smithi106 ceph-mon[25742]: from='mgr.14634 172.21.15.106:0/2720500564' entity='mgr.y' 2024-03-10T00:54:02.339 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:54:02 smithi106 ceph-mon[25742]: pgmap v47: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-10T00:54:02.339 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:54:02 smithi106 ceph-mon[25742]: from='mgr.14634 172.21.15.106:0/2720500564' entity='mgr.y' 2024-03-10T00:54:02.339 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:54:02 smithi106 ceph-mon[25742]: from='mgr.14634 172.21.15.106:0/2720500564' entity='mgr.y' 2024-03-10T00:54:02.524 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:54:02 smithi170 ceph-mon[27620]: from='mgr.14634 172.21.15.106:0/2720500564' entity='mgr.y' 2024-03-10T00:54:02.524 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:54:02 smithi170 ceph-mon[27620]: from='mgr.14634 172.21.15.106:0/2720500564' entity='mgr.y' 2024-03-10T00:54:02.525 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:54:02 smithi170 ceph-mon[27620]: pgmap v47: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-10T00:54:02.525 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:54:02 smithi170 ceph-mon[27620]: from='mgr.14634 172.21.15.106:0/2720500564' entity='mgr.y' 2024-03-10T00:54:02.525 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:54:02 smithi170 ceph-mon[27620]: from='mgr.14634 172.21.15.106:0/2720500564' entity='mgr.y' 2024-03-10T00:54:02.868 INFO:teuthology.orchestra.run.smithi106.stdout:service_type: agent 2024-03-10T00:54:02.868 INFO:teuthology.orchestra.run.smithi106.stdout:service_name: agent 2024-03-10T00:54:02.868 INFO:teuthology.orchestra.run.smithi106.stdout:placement: 2024-03-10T00:54:02.868 INFO:teuthology.orchestra.run.smithi106.stdout: host_pattern: '*' 2024-03-10T00:54:02.868 INFO:teuthology.orchestra.run.smithi106.stdout:status: 2024-03-10T00:54:02.868 INFO:teuthology.orchestra.run.smithi106.stdout: created: '2024-03-10T00:52:37.607414Z' 2024-03-10T00:54:02.868 INFO:teuthology.orchestra.run.smithi106.stdout: last_refresh: '2024-03-10T00:53:59.922174Z' 2024-03-10T00:54:02.868 INFO:teuthology.orchestra.run.smithi106.stdout: running: 2 2024-03-10T00:54:02.868 INFO:teuthology.orchestra.run.smithi106.stdout: size: 2 2024-03-10T00:54:02.868 INFO:teuthology.orchestra.run.smithi106.stdout:events: 2024-03-10T00:54:02.868 INFO:teuthology.orchestra.run.smithi106.stdout:- 2024-03-10T00:52:39.009312Z service:agent [INFO] "service was created" 2024-03-10T00:54:02.869 INFO:teuthology.orchestra.run.smithi106.stdout:--- 2024-03-10T00:54:02.869 INFO:teuthology.orchestra.run.smithi106.stdout:service_type: alertmanager 2024-03-10T00:54:02.869 INFO:teuthology.orchestra.run.smithi106.stdout:service_name: alertmanager 2024-03-10T00:54:02.869 INFO:teuthology.orchestra.run.smithi106.stdout:placement: 2024-03-10T00:54:02.869 INFO:teuthology.orchestra.run.smithi106.stdout: count: 1 2024-03-10T00:54:02.869 INFO:teuthology.orchestra.run.smithi106.stdout: hosts: 2024-03-10T00:54:02.869 INFO:teuthology.orchestra.run.smithi106.stdout: - smithi106=a 2024-03-10T00:54:02.869 INFO:teuthology.orchestra.run.smithi106.stdout:status: 2024-03-10T00:54:02.869 INFO:teuthology.orchestra.run.smithi106.stdout: created: '2024-03-10T00:52:12.807826Z' 2024-03-10T00:54:02.869 INFO:teuthology.orchestra.run.smithi106.stdout: last_refresh: '2024-03-10T00:54:01.774822Z' 2024-03-10T00:54:02.869 INFO:teuthology.orchestra.run.smithi106.stdout: ports: 2024-03-10T00:54:02.869 INFO:teuthology.orchestra.run.smithi106.stdout: - 9093 2024-03-10T00:54:02.869 INFO:teuthology.orchestra.run.smithi106.stdout: - 9094 2024-03-10T00:54:02.869 INFO:teuthology.orchestra.run.smithi106.stdout: running: 1 2024-03-10T00:54:02.870 INFO:teuthology.orchestra.run.smithi106.stdout: size: 1 2024-03-10T00:54:02.870 INFO:teuthology.orchestra.run.smithi106.stdout:events: 2024-03-10T00:54:02.870 INFO:teuthology.orchestra.run.smithi106.stdout:- 2024-03-10T00:53:18.813458Z service:alertmanager [INFO] "service was created" 2024-03-10T00:54:02.870 INFO:teuthology.orchestra.run.smithi106.stdout:--- 2024-03-10T00:54:02.870 INFO:teuthology.orchestra.run.smithi106.stdout:service_type: grafana 2024-03-10T00:54:02.870 INFO:teuthology.orchestra.run.smithi106.stdout:service_name: grafana 2024-03-10T00:54:02.870 INFO:teuthology.orchestra.run.smithi106.stdout:placement: 2024-03-10T00:54:02.870 INFO:teuthology.orchestra.run.smithi106.stdout: count: 1 2024-03-10T00:54:02.870 INFO:teuthology.orchestra.run.smithi106.stdout: hosts: 2024-03-10T00:54:02.870 INFO:teuthology.orchestra.run.smithi106.stdout: - smithi170=a 2024-03-10T00:54:02.870 INFO:teuthology.orchestra.run.smithi106.stdout:spec: 2024-03-10T00:54:02.870 INFO:teuthology.orchestra.run.smithi106.stdout: anonymous_access: true 2024-03-10T00:54:02.870 INFO:teuthology.orchestra.run.smithi106.stdout: protocol: https 2024-03-10T00:54:02.870 INFO:teuthology.orchestra.run.smithi106.stdout:status: 2024-03-10T00:54:02.871 INFO:teuthology.orchestra.run.smithi106.stdout: created: '2024-03-10T00:52:15.840119Z' 2024-03-10T00:54:02.871 INFO:teuthology.orchestra.run.smithi106.stdout: last_refresh: '2024-03-10T00:53:59.923470Z' 2024-03-10T00:54:02.871 INFO:teuthology.orchestra.run.smithi106.stdout: ports: 2024-03-10T00:54:02.871 INFO:teuthology.orchestra.run.smithi106.stdout: - 3000 2024-03-10T00:54:02.871 INFO:teuthology.orchestra.run.smithi106.stdout: running: 1 2024-03-10T00:54:02.871 INFO:teuthology.orchestra.run.smithi106.stdout: size: 1 2024-03-10T00:54:02.871 INFO:teuthology.orchestra.run.smithi106.stdout:events: 2024-03-10T00:54:02.871 INFO:teuthology.orchestra.run.smithi106.stdout:- 2024-03-10T00:53:36.475812Z service:grafana [INFO] "service was created" 2024-03-10T00:54:02.871 INFO:teuthology.orchestra.run.smithi106.stdout:--- 2024-03-10T00:54:02.871 INFO:teuthology.orchestra.run.smithi106.stdout:service_type: iscsi 2024-03-10T00:54:02.871 INFO:teuthology.orchestra.run.smithi106.stdout:service_id: datapool 2024-03-10T00:54:02.871 INFO:teuthology.orchestra.run.smithi106.stdout:service_name: iscsi.datapool 2024-03-10T00:54:02.871 INFO:teuthology.orchestra.run.smithi106.stdout:placement: 2024-03-10T00:54:02.871 INFO:teuthology.orchestra.run.smithi106.stdout: count: 1 2024-03-10T00:54:02.871 INFO:teuthology.orchestra.run.smithi106.stdout: hosts: 2024-03-10T00:54:02.871 INFO:teuthology.orchestra.run.smithi106.stdout: - smithi170=iscsi.a 2024-03-10T00:54:02.871 INFO:teuthology.orchestra.run.smithi106.stdout:spec: 2024-03-10T00:54:02.872 INFO:teuthology.orchestra.run.smithi106.stdout: api_password: admin 2024-03-10T00:54:02.872 INFO:teuthology.orchestra.run.smithi106.stdout: api_port: 5000 2024-03-10T00:54:02.872 INFO:teuthology.orchestra.run.smithi106.stdout: api_user: admin 2024-03-10T00:54:02.872 INFO:teuthology.orchestra.run.smithi106.stdout: pool: datapool 2024-03-10T00:54:02.872 INFO:teuthology.orchestra.run.smithi106.stdout: trusted_ip_list: 172.21.15.170 2024-03-10T00:54:02.872 INFO:teuthology.orchestra.run.smithi106.stdout:status: 2024-03-10T00:54:02.872 INFO:teuthology.orchestra.run.smithi106.stdout: created: '2024-03-10T00:52:00.249677Z' 2024-03-10T00:54:02.872 INFO:teuthology.orchestra.run.smithi106.stdout: last_refresh: '2024-03-10T00:53:59.923259Z' 2024-03-10T00:54:02.872 INFO:teuthology.orchestra.run.smithi106.stdout: ports: 2024-03-10T00:54:02.872 INFO:teuthology.orchestra.run.smithi106.stdout: - 5000 2024-03-10T00:54:02.872 INFO:teuthology.orchestra.run.smithi106.stdout: running: 1 2024-03-10T00:54:02.872 INFO:teuthology.orchestra.run.smithi106.stdout: size: 1 2024-03-10T00:54:02.872 INFO:teuthology.orchestra.run.smithi106.stdout:--- 2024-03-10T00:54:02.872 INFO:teuthology.orchestra.run.smithi106.stdout:service_type: mgr 2024-03-10T00:54:02.872 INFO:teuthology.orchestra.run.smithi106.stdout:service_name: mgr 2024-03-10T00:54:02.872 INFO:teuthology.orchestra.run.smithi106.stdout:placement: 2024-03-10T00:54:02.873 INFO:teuthology.orchestra.run.smithi106.stdout: count: 2 2024-03-10T00:54:02.873 INFO:teuthology.orchestra.run.smithi106.stdout: hosts: 2024-03-10T00:54:02.873 INFO:teuthology.orchestra.run.smithi106.stdout: - smithi106=y 2024-03-10T00:54:02.873 INFO:teuthology.orchestra.run.smithi106.stdout: - smithi170=x 2024-03-10T00:54:02.873 INFO:teuthology.orchestra.run.smithi106.stdout:status: 2024-03-10T00:54:02.873 INFO:teuthology.orchestra.run.smithi106.stdout: created: '2024-03-10T00:48:01.788518Z' 2024-03-10T00:54:02.873 INFO:teuthology.orchestra.run.smithi106.stdout: last_refresh: '2024-03-10T00:53:59.922526Z' 2024-03-10T00:54:02.873 INFO:teuthology.orchestra.run.smithi106.stdout: running: 2 2024-03-10T00:54:02.873 INFO:teuthology.orchestra.run.smithi106.stdout: size: 2 2024-03-10T00:54:02.873 INFO:teuthology.orchestra.run.smithi106.stdout:--- 2024-03-10T00:54:02.873 INFO:teuthology.orchestra.run.smithi106.stdout:service_type: mon 2024-03-10T00:54:02.873 INFO:teuthology.orchestra.run.smithi106.stdout:service_name: mon 2024-03-10T00:54:02.873 INFO:teuthology.orchestra.run.smithi106.stdout:placement: 2024-03-10T00:54:02.873 INFO:teuthology.orchestra.run.smithi106.stdout: count: 3 2024-03-10T00:54:02.873 INFO:teuthology.orchestra.run.smithi106.stdout: hosts: 2024-03-10T00:54:02.873 INFO:teuthology.orchestra.run.smithi106.stdout: - smithi106:172.21.15.106=a 2024-03-10T00:54:02.874 INFO:teuthology.orchestra.run.smithi106.stdout: - smithi106:[v2:172.21.15.106:3301,v1:172.21.15.106:6790]=c 2024-03-10T00:54:02.874 INFO:teuthology.orchestra.run.smithi106.stdout: - smithi170:172.21.15.170=b 2024-03-10T00:54:02.874 INFO:teuthology.orchestra.run.smithi106.stdout:status: 2024-03-10T00:54:02.874 INFO:teuthology.orchestra.run.smithi106.stdout: created: '2024-03-10T00:47:32.456283Z' 2024-03-10T00:54:02.874 INFO:teuthology.orchestra.run.smithi106.stdout: last_refresh: '2024-03-10T00:53:59.922365Z' 2024-03-10T00:54:02.874 INFO:teuthology.orchestra.run.smithi106.stdout: running: 3 2024-03-10T00:54:02.874 INFO:teuthology.orchestra.run.smithi106.stdout: size: 3 2024-03-10T00:54:02.874 INFO:teuthology.orchestra.run.smithi106.stdout:--- 2024-03-10T00:54:02.874 INFO:teuthology.orchestra.run.smithi106.stdout:service_type: node-exporter 2024-03-10T00:54:02.874 INFO:teuthology.orchestra.run.smithi106.stdout:service_name: node-exporter 2024-03-10T00:54:02.874 INFO:teuthology.orchestra.run.smithi106.stdout:placement: 2024-03-10T00:54:02.874 INFO:teuthology.orchestra.run.smithi106.stdout: count: 2 2024-03-10T00:54:02.874 INFO:teuthology.orchestra.run.smithi106.stdout: hosts: 2024-03-10T00:54:02.874 INFO:teuthology.orchestra.run.smithi106.stdout: - smithi106=a 2024-03-10T00:54:02.874 INFO:teuthology.orchestra.run.smithi106.stdout: - smithi170=b 2024-03-10T00:54:02.875 INFO:teuthology.orchestra.run.smithi106.stdout:status: 2024-03-10T00:54:02.875 INFO:teuthology.orchestra.run.smithi106.stdout: created: '2024-03-10T00:52:08.963194Z' 2024-03-10T00:54:02.875 INFO:teuthology.orchestra.run.smithi106.stdout: last_refresh: '2024-03-10T00:53:59.923545Z' 2024-03-10T00:54:02.875 INFO:teuthology.orchestra.run.smithi106.stdout: ports: 2024-03-10T00:54:02.875 INFO:teuthology.orchestra.run.smithi106.stdout: - 9100 2024-03-10T00:54:02.875 INFO:teuthology.orchestra.run.smithi106.stdout: running: 2 2024-03-10T00:54:02.875 INFO:teuthology.orchestra.run.smithi106.stdout: size: 2 2024-03-10T00:54:02.875 INFO:teuthology.orchestra.run.smithi106.stdout:events: 2024-03-10T00:54:02.875 INFO:teuthology.orchestra.run.smithi106.stdout:- 2024-03-10T00:53:48.391013Z service:node-exporter [INFO] "service was created" 2024-03-10T00:54:02.875 INFO:teuthology.orchestra.run.smithi106.stdout:--- 2024-03-10T00:54:02.875 INFO:teuthology.orchestra.run.smithi106.stdout:service_type: osd 2024-03-10T00:54:02.875 INFO:teuthology.orchestra.run.smithi106.stdout:service_name: osd 2024-03-10T00:54:02.875 INFO:teuthology.orchestra.run.smithi106.stdout:unmanaged: true 2024-03-10T00:54:02.876 INFO:teuthology.orchestra.run.smithi106.stdout:spec: 2024-03-10T00:54:02.876 INFO:teuthology.orchestra.run.smithi106.stdout: filter_logic: AND 2024-03-10T00:54:02.876 INFO:teuthology.orchestra.run.smithi106.stdout: objectstore: bluestore 2024-03-10T00:54:02.876 INFO:teuthology.orchestra.run.smithi106.stdout:status: 2024-03-10T00:54:02.876 INFO:teuthology.orchestra.run.smithi106.stdout: container_image_id: 01a4f6a6bb3e92ab9ab39a2f835684ea9b10e6ec9fb81ac08ebea4165583a538 2024-03-10T00:54:02.876 INFO:teuthology.orchestra.run.smithi106.stdout: container_image_name: quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bc4a24690ce5d7255d490e21058d8c114b55e1b93a3d8345f90b6700ad29dada 2024-03-10T00:54:02.876 INFO:teuthology.orchestra.run.smithi106.stdout: last_refresh: '2024-03-10T00:53:59.922680Z' 2024-03-10T00:54:02.876 INFO:teuthology.orchestra.run.smithi106.stdout: running: 8 2024-03-10T00:54:02.876 INFO:teuthology.orchestra.run.smithi106.stdout: size: 8 2024-03-10T00:54:02.876 INFO:teuthology.orchestra.run.smithi106.stdout:--- 2024-03-10T00:54:02.876 INFO:teuthology.orchestra.run.smithi106.stdout:service_type: prometheus 2024-03-10T00:54:02.876 INFO:teuthology.orchestra.run.smithi106.stdout:service_name: prometheus 2024-03-10T00:54:02.876 INFO:teuthology.orchestra.run.smithi106.stdout:placement: 2024-03-10T00:54:02.876 INFO:teuthology.orchestra.run.smithi106.stdout: count: 1 2024-03-10T00:54:02.876 INFO:teuthology.orchestra.run.smithi106.stdout: hosts: 2024-03-10T00:54:02.876 INFO:teuthology.orchestra.run.smithi106.stdout: - smithi170=a 2024-03-10T00:54:02.877 INFO:teuthology.orchestra.run.smithi106.stdout:status: 2024-03-10T00:54:02.877 INFO:teuthology.orchestra.run.smithi106.stdout: created: '2024-03-10T00:52:04.275654Z' 2024-03-10T00:54:02.877 INFO:teuthology.orchestra.run.smithi106.stdout: last_refresh: '2024-03-10T00:53:59.923389Z' 2024-03-10T00:54:02.877 INFO:teuthology.orchestra.run.smithi106.stdout: ports: 2024-03-10T00:54:02.877 INFO:teuthology.orchestra.run.smithi106.stdout: - 9095 2024-03-10T00:54:02.877 INFO:teuthology.orchestra.run.smithi106.stdout: running: 1 2024-03-10T00:54:02.877 INFO:teuthology.orchestra.run.smithi106.stdout: size: 1 2024-03-10T00:54:02.877 INFO:teuthology.orchestra.run.smithi106.stdout:events: 2024-03-10T00:54:02.877 INFO:teuthology.orchestra.run.smithi106.stdout:- 2024-03-10T00:53:48.400767Z service:prometheus [INFO] "service was created" 2024-03-10T00:54:02.877 INFO:teuthology.orchestra.run.smithi106.stdout:--- 2024-03-10T00:54:02.877 INFO:teuthology.orchestra.run.smithi106.stdout:service_type: rgw 2024-03-10T00:54:02.877 INFO:teuthology.orchestra.run.smithi106.stdout:service_id: foo.a 2024-03-10T00:54:02.877 INFO:teuthology.orchestra.run.smithi106.stdout:service_name: rgw.foo.a 2024-03-10T00:54:02.877 INFO:teuthology.orchestra.run.smithi106.stdout:placement: 2024-03-10T00:54:02.877 INFO:teuthology.orchestra.run.smithi106.stdout: count: 1 2024-03-10T00:54:02.877 INFO:teuthology.orchestra.run.smithi106.stdout: hosts: 2024-03-10T00:54:02.878 INFO:teuthology.orchestra.run.smithi106.stdout: - smithi106=foo.a 2024-03-10T00:54:02.878 INFO:teuthology.orchestra.run.smithi106.stdout:status: 2024-03-10T00:54:02.878 INFO:teuthology.orchestra.run.smithi106.stdout: created: '2024-03-10T00:51:52.133695Z' 2024-03-10T00:54:02.878 INFO:teuthology.orchestra.run.smithi106.stdout: last_refresh: '2024-03-10T00:54:01.774751Z' 2024-03-10T00:54:02.878 INFO:teuthology.orchestra.run.smithi106.stdout: ports: 2024-03-10T00:54:02.878 INFO:teuthology.orchestra.run.smithi106.stdout: - 80 2024-03-10T00:54:02.878 INFO:teuthology.orchestra.run.smithi106.stdout: running: 1 2024-03-10T00:54:02.878 INFO:teuthology.orchestra.run.smithi106.stdout: size: 1 2024-03-10T00:54:03.704 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:831bc188fa69f143bbba69ac76119e784500e8e3 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 75c37498-de77-11ee-95c7-87774f69a715 -- bash -c 'ceph orch ls | grep '"'"'^osd '"'"'' 2024-03-10T00:54:03.926 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:54:03 smithi106 ceph-mon[33460]: from='mgr.14634 172.21.15.106:0/2720500564' entity='mgr.y' 2024-03-10T00:54:03.927 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:54:03 smithi106 ceph-mon[33460]: from='mgr.14634 172.21.15.106:0/2720500564' entity='mgr.y' 2024-03-10T00:54:03.927 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:54:03 smithi106 ceph-mon[33460]: from='client.24704 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "yaml"}]: dispatch 2024-03-10T00:54:03.927 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:54:03 smithi106 ceph-mon[25742]: from='mgr.14634 172.21.15.106:0/2720500564' entity='mgr.y' 2024-03-10T00:54:03.927 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:54:03 smithi106 ceph-mon[25742]: from='mgr.14634 172.21.15.106:0/2720500564' entity='mgr.y' 2024-03-10T00:54:03.927 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:54:03 smithi106 ceph-mon[25742]: from='client.24704 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "yaml"}]: dispatch 2024-03-10T00:54:04.024 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:54:03 smithi170 ceph-mon[27620]: from='mgr.14634 172.21.15.106:0/2720500564' entity='mgr.y' 2024-03-10T00:54:04.024 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:54:03 smithi170 ceph-mon[27620]: from='mgr.14634 172.21.15.106:0/2720500564' entity='mgr.y' 2024-03-10T00:54:04.024 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:54:03 smithi170 ceph-mon[27620]: from='client.24704 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "yaml"}]: dispatch 2024-03-10T00:54:04.531 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/75c37498-de77-11ee-95c7-87774f69a715/mon.a/config 2024-03-10T00:54:04.926 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:54:04 smithi106 ceph-mon[25742]: pgmap v48: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-10T00:54:04.926 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:54:04 smithi106 ceph-mon[33460]: pgmap v48: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-10T00:54:05.275 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:54:04 smithi170 ceph-mon[27620]: pgmap v48: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-10T00:54:06.649 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:54:06 smithi170 ceph-mon[27620]: pgmap v49: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-10T00:54:06.717 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:54:06 smithi106 ceph-mon[33460]: pgmap v49: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-10T00:54:06.718 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:54:06 smithi106 ceph-mon[25742]: pgmap v49: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-10T00:54:07.358 INFO:teuthology.orchestra.run.smithi106.stdout:osd 8 7s ago - 2024-03-10T00:54:08.034 DEBUG:teuthology.run_tasks:Unwinding manager cephadm 2024-03-10T00:54:08.043 INFO:tasks.cephadm:Teardown begin 2024-03-10T00:54:08.043 DEBUG:teuthology.orchestra.run.smithi106:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-03-10T00:54:08.073 DEBUG:teuthology.orchestra.run.smithi170:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-03-10T00:54:08.109 INFO:tasks.cephadm:Cleaning up testdir ceph.* files... 2024-03-10T00:54:08.109 DEBUG:teuthology.orchestra.run.smithi106:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-03-10T00:54:08.131 DEBUG:teuthology.orchestra.run.smithi170:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-03-10T00:54:08.166 INFO:tasks.cephadm:Stopping all daemons... 2024-03-10T00:54:08.166 INFO:tasks.cephadm.mon.a:Stopping mon.a... 2024-03-10T00:54:08.167 DEBUG:teuthology.orchestra.run.smithi106:> sudo systemctl stop ceph-75c37498-de77-11ee-95c7-87774f69a715@mon.a 2024-03-10T00:54:08.338 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:54:08 smithi106 systemd[1]: Stopping Ceph mon.a for 75c37498-de77-11ee-95c7-87774f69a715... 2024-03-10T00:54:08.774 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:54:08 smithi170 ceph-mon[27620]: from='client.14820 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-10T00:54:08.774 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:54:08 smithi170 ceph-mon[27620]: pgmap v50: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-10T00:54:08.774 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:54:08 smithi170 ceph-mon[27620]: from='mgr.14634 172.21.15.106:0/2720500564' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-10T00:54:08.775 INFO:journalctl@ceph.iscsi.iscsi.a.smithi170.stdout:Mar 10 00:54:08 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-iscsi-iscsi-a[50134]: debug there is no tcmu-runner data available 2024-03-10T00:54:08.909 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:54:08 smithi106 ceph-mon[33460]: from='client.14820 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-10T00:54:08.909 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:54:08 smithi106 ceph-mon[33460]: pgmap v50: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-10T00:54:08.909 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:54:08 smithi106 ceph-mon[33460]: from='mgr.14634 172.21.15.106:0/2720500564' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-10T00:54:08.909 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:54:08 smithi106 ceph-mon[25742]: from='client.14820 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-10T00:54:08.910 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:54:08 smithi106 ceph-mon[25742]: pgmap v50: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-10T00:54:08.910 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:54:08 smithi106 ceph-mon[25742]: from='mgr.14634 172.21.15.106:0/2720500564' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-10T00:54:08.910 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:54:08 smithi106 ceph-75c37498-de77-11ee-95c7-87774f69a715-mon-a[25719]: 2024-03-10T00:54:08.859+0000 7f80e402e700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-03-10T00:54:08.910 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:54:08 smithi106 ceph-75c37498-de77-11ee-95c7-87774f69a715-mon-a[25719]: 2024-03-10T00:54:08.859+0000 7f80e402e700 -1 mon.a@0(leader) e3 *** Got Signal Terminated *** 2024-03-10T00:54:08.910 INFO:journalctl@ceph.mgr.y.smithi106.stdout:Mar 10 00:54:08 smithi106 ceph-75c37498-de77-11ee-95c7-87774f69a715-mgr-y[25997]: [10/Mar/2024:00:54:08] ENGINE Bus STOPPING 2024-03-10T00:54:08.910 INFO:journalctl@ceph.mgr.y.smithi106.stdout:Mar 10 00:54:08 smithi106 ceph-75c37498-de77-11ee-95c7-87774f69a715-mgr-y[25997]: [10/Mar/2024:00:54:08] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-03-10T00:54:08.911 INFO:journalctl@ceph.mgr.y.smithi106.stdout:Mar 10 00:54:08 smithi106 ceph-75c37498-de77-11ee-95c7-87774f69a715-mgr-y[25997]: [10/Mar/2024:00:54:08] ENGINE Bus STOPPED 2024-03-10T00:54:08.911 INFO:journalctl@ceph.mgr.y.smithi106.stdout:Mar 10 00:54:08 smithi106 ceph-75c37498-de77-11ee-95c7-87774f69a715-mgr-y[25997]: [10/Mar/2024:00:54:08] ENGINE Bus STARTING 2024-03-10T00:54:09.232 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:54:08 smithi106 podman[76902]: 2024-03-10 00:54:08.912187969 +0000 UTC m=+0.172260142 container died 97a03d7ed61790350cb5cb41bd58fe8d1f02ce8a94e85b2c0994a6ae2a89679b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:831bc188fa69f143bbba69ac76119e784500e8e3, name=ceph-75c37498-de77-11ee-95c7-87774f69a715-mon-a, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, RELEASE=wip-yuri4-testing-2024-03-05-0854-831bc18, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.build-date=20240222) 2024-03-10T00:54:09.232 INFO:journalctl@ceph.mgr.y.smithi106.stdout:Mar 10 00:54:08 smithi106 ceph-75c37498-de77-11ee-95c7-87774f69a715-mgr-y[25997]: [10/Mar/2024:00:54:08] ENGINE Serving on http://:::9283 2024-03-10T00:54:09.232 INFO:journalctl@ceph.mgr.y.smithi106.stdout:Mar 10 00:54:08 smithi106 ceph-75c37498-de77-11ee-95c7-87774f69a715-mgr-y[25997]: [10/Mar/2024:00:54:08] ENGINE Bus STARTED 2024-03-10T00:54:09.588 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:54:09 smithi106 podman[76902]: 2024-03-10 00:54:09.400251518 +0000 UTC m=+0.660323698 container cleanup 97a03d7ed61790350cb5cb41bd58fe8d1f02ce8a94e85b2c0994a6ae2a89679b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:831bc188fa69f143bbba69ac76119e784500e8e3, name=ceph-75c37498-de77-11ee-95c7-87774f69a715-mon-a, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, ceph=True, org.label-schema.build-date=20240222, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, RELEASE=wip-yuri4-testing-2024-03-05-0854-831bc18, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5) 2024-03-10T00:54:09.588 INFO:journalctl@ceph.mon.a.smithi106.stdout:Mar 10 00:54:09 smithi106 bash[76902]: ceph-75c37498-de77-11ee-95c7-87774f69a715-mon-a 2024-03-10T00:54:09.827 DEBUG:teuthology.orchestra.run.smithi106:> sudo pkill -f 'journalctl -f -n 0 -u ceph-75c37498-de77-11ee-95c7-87774f69a715@mon.a.service' 2024-03-10T00:54:09.878 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-10T00:54:09.878 INFO:tasks.cephadm.mon.a:Stopped mon.a 2024-03-10T00:54:09.879 INFO:tasks.cephadm.mon.b:Stopping mon.c... 2024-03-10T00:54:09.879 DEBUG:teuthology.orchestra.run.smithi106:> sudo systemctl stop ceph-75c37498-de77-11ee-95c7-87774f69a715@mon.c 2024-03-10T00:54:10.088 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:54:09 smithi106 systemd[1]: Stopping Ceph mon.c for 75c37498-de77-11ee-95c7-87774f69a715... 2024-03-10T00:54:10.462 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:54:10 smithi106 ceph-75c37498-de77-11ee-95c7-87774f69a715-mon-c[33437]: 2024-03-10T00:54:10.197+0000 7fbe50fd7700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.c -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-03-10T00:54:10.462 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:54:10 smithi106 ceph-75c37498-de77-11ee-95c7-87774f69a715-mon-c[33437]: 2024-03-10T00:54:10.197+0000 7fbe50fd7700 -1 mon.c@2(peon) e3 *** Got Signal Terminated *** 2024-03-10T00:54:10.744 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:54:10 smithi106 podman[77043]: 2024-03-10 00:54:10.465138425 +0000 UTC m=+0.395090663 container died 2411e393082d5ae1f677c7a188bf2ee903ed532e4f6db2cd90cc776ace2c1493 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bc4a24690ce5d7255d490e21058d8c114b55e1b93a3d8345f90b6700ad29dada, name=ceph-75c37498-de77-11ee-95c7-87774f69a715-mon-c, RELEASE=wip-yuri4-testing-2024-03-05-0854-831bc18, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.build-date=20240222, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS) 2024-03-10T00:54:11.041 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:54:10 smithi106 podman[77043]: 2024-03-10 00:54:10.921205461 +0000 UTC m=+0.851157707 container cleanup 2411e393082d5ae1f677c7a188bf2ee903ed532e4f6db2cd90cc776ace2c1493 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bc4a24690ce5d7255d490e21058d8c114b55e1b93a3d8345f90b6700ad29dada, name=ceph-75c37498-de77-11ee-95c7-87774f69a715-mon-c, io.buildah.version=1.33.5, org.label-schema.build-date=20240222, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_CLEAN=True, RELEASE=wip-yuri4-testing-2024-03-05-0854-831bc18) 2024-03-10T00:54:11.042 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:54:10 smithi106 bash[77043]: ceph-75c37498-de77-11ee-95c7-87774f69a715-mon-c 2024-03-10T00:54:11.764 INFO:journalctl@ceph.mgr.y.smithi106.stdout:Mar 10 00:54:11 smithi106 ceph-75c37498-de77-11ee-95c7-87774f69a715-mgr-y[25997]: ::ffff:172.21.15.170 - - [10/Mar/2024:00:54:11] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-10T00:54:11.764 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:54:11 smithi106 podman[77062]: 2024-03-10 00:54:11.449423029 +0000 UTC m=+0.982214946 container remove 2411e393082d5ae1f677c7a188bf2ee903ed532e4f6db2cd90cc776ace2c1493 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bc4a24690ce5d7255d490e21058d8c114b55e1b93a3d8345f90b6700ad29dada, name=ceph-75c37498-de77-11ee-95c7-87774f69a715-mon-c, ceph=True, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, RELEASE=wip-yuri4-testing-2024-03-05-0854-831bc18, org.label-schema.build-date=20240222, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-10T00:54:11.999 DEBUG:teuthology.orchestra.run.smithi106:> sudo pkill -f 'journalctl -f -n 0 -u ceph-75c37498-de77-11ee-95c7-87774f69a715@mon.c.service' 2024-03-10T00:54:12.024 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:54:11 smithi106 systemd[1]: ceph-75c37498-de77-11ee-95c7-87774f69a715@mon.c.service: Deactivated successfully. 2024-03-10T00:54:12.024 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:54:11 smithi106 systemd[1]: Stopped Ceph mon.c for 75c37498-de77-11ee-95c7-87774f69a715. 2024-03-10T00:54:12.024 INFO:journalctl@ceph.mon.c.smithi106.stdout:Mar 10 00:54:11 smithi106 systemd[1]: ceph-75c37498-de77-11ee-95c7-87774f69a715@mon.c.service: Consumed 4.640s CPU time. 2024-03-10T00:54:12.041 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-10T00:54:12.041 INFO:tasks.cephadm.mon.b:Stopped mon.c 2024-03-10T00:54:12.041 INFO:tasks.cephadm.mon.b:Stopping mon.b... 2024-03-10T00:54:12.041 DEBUG:teuthology.orchestra.run.smithi170:> sudo systemctl stop ceph-75c37498-de77-11ee-95c7-87774f69a715@mon.b 2024-03-10T00:54:12.204 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:54:12 smithi170 systemd[1]: Stopping Ceph mon.b for 75c37498-de77-11ee-95c7-87774f69a715... 2024-03-10T00:54:12.891 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:54:12 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-mon-b[27596]: 2024-03-10T00:54:12.565+0000 7f74079a9700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.b -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-03-10T00:54:12.891 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:54:12 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-mon-b[27596]: 2024-03-10T00:54:12.566+0000 7f74079a9700 -1 mon.b@1(peon) e3 *** Got Signal Terminated *** 2024-03-10T00:54:12.891 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:54:12 smithi170 podman[64601]: 2024-03-10 00:54:12.586902968 +0000 UTC m=+0.121128683 container died 4883b5f0e4191157e18a4724a34785eeba6ba990ee71d8e3351825484ed9079d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bc4a24690ce5d7255d490e21058d8c114b55e1b93a3d8345f90b6700ad29dada, name=ceph-75c37498-de77-11ee-95c7-87774f69a715-mon-b, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, GIT_CLEAN=True, RELEASE=wip-yuri4-testing-2024-03-05-0854-831bc18, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.build-date=20240222, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, maintainer=Guillaume Abrioux ) 2024-03-10T00:54:13.143 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:54:13 smithi170 podman[64601]: 2024-03-10 00:54:13.001946193 +0000 UTC m=+0.536171922 container cleanup 4883b5f0e4191157e18a4724a34785eeba6ba990ee71d8e3351825484ed9079d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bc4a24690ce5d7255d490e21058d8c114b55e1b93a3d8345f90b6700ad29dada, name=ceph-75c37498-de77-11ee-95c7-87774f69a715-mon-b, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, ceph=True, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, GIT_CLEAN=True, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.build-date=20240222, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-yuri4-testing-2024-03-05-0854-831bc18) 2024-03-10T00:54:13.143 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:54:13 smithi170 bash[64601]: ceph-75c37498-de77-11ee-95c7-87774f69a715-mon-b 2024-03-10T00:54:13.774 INFO:journalctl@ceph.mon.b.smithi170.stdout:Mar 10 00:54:13 smithi170 podman[64625]: 2024-03-10 00:54:13.496721444 +0000 UTC m=+0.909588423 container remove 4883b5f0e4191157e18a4724a34785eeba6ba990ee71d8e3351825484ed9079d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bc4a24690ce5d7255d490e21058d8c114b55e1b93a3d8345f90b6700ad29dada, name=ceph-75c37498-de77-11ee-95c7-87774f69a715-mon-b, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240222, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, io.buildah.version=1.33.5, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, ceph=True, RELEASE=wip-yuri4-testing-2024-03-05-0854-831bc18, maintainer=Guillaume Abrioux ) 2024-03-10T00:54:14.061 DEBUG:teuthology.orchestra.run.smithi170:> sudo pkill -f 'journalctl -f -n 0 -u ceph-75c37498-de77-11ee-95c7-87774f69a715@mon.b.service' 2024-03-10T00:54:14.110 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-10T00:54:14.110 INFO:tasks.cephadm.mon.b:Stopped mon.b 2024-03-10T00:54:14.110 INFO:tasks.cephadm.mgr.y:Stopping mgr.y... 2024-03-10T00:54:14.110 DEBUG:teuthology.orchestra.run.smithi106:> sudo systemctl stop ceph-75c37498-de77-11ee-95c7-87774f69a715@mgr.y 2024-03-10T00:54:14.333 INFO:journalctl@ceph.mgr.y.smithi106.stdout:Mar 10 00:54:14 smithi106 systemd[1]: Stopping Ceph mgr.y for 75c37498-de77-11ee-95c7-87774f69a715... 2024-03-10T00:54:15.088 INFO:journalctl@ceph.mgr.y.smithi106.stdout:Mar 10 00:54:15 smithi106 ceph-75c37498-de77-11ee-95c7-87774f69a715-mgr-y[25997]: 2024-03-10T00:54:15.018+0000 7fcffc235700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mgr -n mgr.y -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-10T00:54:15.088 INFO:journalctl@ceph.mgr.y.smithi106.stdout:Mar 10 00:54:15 smithi106 ceph-75c37498-de77-11ee-95c7-87774f69a715-mgr-y[25997]: 2024-03-10T00:54:15.018+0000 7fcffc235700 -1 mgr handle_mgr_signal *** Got signal Terminated *** 2024-03-10T00:54:15.088 INFO:journalctl@ceph.mgr.y.smithi106.stdout:Mar 10 00:54:15 smithi106 podman[77592]: 2024-03-10 00:54:15.051323702 +0000 UTC m=+0.141634762 container died 52b050b0e011b0de5eaf2a09f03bb853179ca77c4652d1e8ff867cc0e4013ab7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:831bc188fa69f143bbba69ac76119e784500e8e3, name=ceph-75c37498-de77-11ee-95c7-87774f69a715-mgr-y, ceph=True, maintainer=Guillaume Abrioux , RELEASE=wip-yuri4-testing-2024-03-05-0854-831bc18, io.buildah.version=1.33.5, org.label-schema.build-date=20240222, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_CLEAN=True) 2024-03-10T00:54:15.681 INFO:journalctl@ceph.mgr.y.smithi106.stdout:Mar 10 00:54:15 smithi106 podman[77592]: 2024-03-10 00:54:15.416403888 +0000 UTC m=+0.506714946 container cleanup 52b050b0e011b0de5eaf2a09f03bb853179ca77c4652d1e8ff867cc0e4013ab7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:831bc188fa69f143bbba69ac76119e784500e8e3, name=ceph-75c37498-de77-11ee-95c7-87774f69a715-mgr-y, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.build-date=20240222, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, RELEASE=wip-yuri4-testing-2024-03-05-0854-831bc18, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True) 2024-03-10T00:54:15.681 INFO:journalctl@ceph.mgr.y.smithi106.stdout:Mar 10 00:54:15 smithi106 bash[77592]: ceph-75c37498-de77-11ee-95c7-87774f69a715-mgr-y 2024-03-10T00:54:15.833 DEBUG:teuthology.orchestra.run.smithi106:> sudo pkill -f 'journalctl -f -n 0 -u ceph-75c37498-de77-11ee-95c7-87774f69a715@mgr.y.service' 2024-03-10T00:54:15.876 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-10T00:54:15.876 INFO:tasks.cephadm.mgr.y:Stopped mgr.y 2024-03-10T00:54:15.877 INFO:tasks.cephadm.mgr.x:Stopping mgr.x... 2024-03-10T00:54:15.877 DEBUG:teuthology.orchestra.run.smithi170:> sudo systemctl stop ceph-75c37498-de77-11ee-95c7-87774f69a715@mgr.x 2024-03-10T00:54:16.215 INFO:journalctl@ceph.mgr.x.smithi170.stdout:Mar 10 00:54:15 smithi170 systemd[1]: Stopping Ceph mgr.x for 75c37498-de77-11ee-95c7-87774f69a715... 2024-03-10T00:54:16.524 INFO:journalctl@ceph.mgr.x.smithi170.stdout:Mar 10 00:54:16 smithi170 podman[64818]: 2024-03-10 00:54:16.218133675 +0000 UTC m=+0.147373816 container died 5fd8a349b80c9a6877bf39c8317aaf7eb359bbe9845db773e80dca74a2b4cfaf (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bc4a24690ce5d7255d490e21058d8c114b55e1b93a3d8345f90b6700ad29dada, name=ceph-75c37498-de77-11ee-95c7-87774f69a715-mgr-x, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, RELEASE=wip-yuri4-testing-2024-03-05-0854-831bc18, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.build-date=20240222, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=) 2024-03-10T00:54:16.818 INFO:journalctl@ceph.mgr.x.smithi170.stdout:Mar 10 00:54:16 smithi170 podman[64818]: 2024-03-10 00:54:16.569612606 +0000 UTC m=+0.498852747 container cleanup 5fd8a349b80c9a6877bf39c8317aaf7eb359bbe9845db773e80dca74a2b4cfaf (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bc4a24690ce5d7255d490e21058d8c114b55e1b93a3d8345f90b6700ad29dada, name=ceph-75c37498-de77-11ee-95c7-87774f69a715-mgr-x, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, ceph=True, RELEASE=wip-yuri4-testing-2024-03-05-0854-831bc18, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240222, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS) 2024-03-10T00:54:16.818 INFO:journalctl@ceph.mgr.x.smithi170.stdout:Mar 10 00:54:16 smithi170 bash[64818]: ceph-75c37498-de77-11ee-95c7-87774f69a715-mgr-x 2024-03-10T00:54:16.974 DEBUG:teuthology.orchestra.run.smithi170:> sudo pkill -f 'journalctl -f -n 0 -u ceph-75c37498-de77-11ee-95c7-87774f69a715@mgr.x.service' 2024-03-10T00:54:17.012 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-10T00:54:17.012 INFO:tasks.cephadm.mgr.x:Stopped mgr.x 2024-03-10T00:54:17.013 INFO:tasks.cephadm.osd.0:Stopping osd.0... 2024-03-10T00:54:17.013 DEBUG:teuthology.orchestra.run.smithi106:> sudo systemctl stop ceph-75c37498-de77-11ee-95c7-87774f69a715@osd.0 2024-03-10T00:54:17.338 INFO:journalctl@ceph.osd.0.smithi106.stdout:Mar 10 00:54:17 smithi106 systemd[1]: Stopping Ceph osd.0 for 75c37498-de77-11ee-95c7-87774f69a715... 2024-03-10T00:54:17.838 INFO:journalctl@ceph.osd.0.smithi106.stdout:Mar 10 00:54:17 smithi106 ceph-75c37498-de77-11ee-95c7-87774f69a715-osd-0[38522]: 2024-03-10T00:54:17.654+0000 7fc49e575700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.0 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-10T00:54:17.838 INFO:journalctl@ceph.osd.0.smithi106.stdout:Mar 10 00:54:17 smithi106 ceph-75c37498-de77-11ee-95c7-87774f69a715-osd-0[38522]: 2024-03-10T00:54:17.654+0000 7fc49e575700 -1 osd.0 71 *** Got signal Terminated *** 2024-03-10T00:54:17.838 INFO:journalctl@ceph.osd.0.smithi106.stdout:Mar 10 00:54:17 smithi106 ceph-75c37498-de77-11ee-95c7-87774f69a715-osd-0[38522]: 2024-03-10T00:54:17.654+0000 7fc49e575700 -1 osd.0 71 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-10T00:54:23.283 INFO:journalctl@ceph.osd.0.smithi106.stdout:Mar 10 00:54:23 smithi106 podman[78028]: 2024-03-10 00:54:23.000408072 +0000 UTC m=+5.800553598 container died 80d749660f4d575d11470f9c16d396fc78fd8417147624fd9940740c153430bf (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bc4a24690ce5d7255d490e21058d8c114b55e1b93a3d8345f90b6700ad29dada, name=ceph-75c37498-de77-11ee-95c7-87774f69a715-osd-0, io.buildah.version=1.33.5, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_CLEAN=True, maintainer=Guillaume Abrioux , RELEASE=wip-yuri4-testing-2024-03-05-0854-831bc18, org.label-schema.build-date=20240222, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD) 2024-03-10T00:54:23.587 INFO:journalctl@ceph.osd.0.smithi106.stdout:Mar 10 00:54:23 smithi106 podman[78028]: 2024-03-10 00:54:23.434857173 +0000 UTC m=+6.235002694 container cleanup 80d749660f4d575d11470f9c16d396fc78fd8417147624fd9940740c153430bf (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bc4a24690ce5d7255d490e21058d8c114b55e1b93a3d8345f90b6700ad29dada, name=ceph-75c37498-de77-11ee-95c7-87774f69a715-osd-0, RELEASE=wip-yuri4-testing-2024-03-05-0854-831bc18, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.build-date=20240222, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD) 2024-03-10T00:54:23.588 INFO:journalctl@ceph.osd.0.smithi106.stdout:Mar 10 00:54:23 smithi106 bash[78028]: ceph-75c37498-de77-11ee-95c7-87774f69a715-osd-0 2024-03-10T00:54:24.049 INFO:journalctl@ceph.osd.0.smithi106.stdout:Mar 10 00:54:23 smithi106 podman[78088]: 2024-03-10 00:54:23.629640486 +0000 UTC m=+0.629047400 container remove 80d749660f4d575d11470f9c16d396fc78fd8417147624fd9940740c153430bf (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bc4a24690ce5d7255d490e21058d8c114b55e1b93a3d8345f90b6700ad29dada, name=ceph-75c37498-de77-11ee-95c7-87774f69a715-osd-0, RELEASE=wip-yuri4-testing-2024-03-05-0854-831bc18, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240222, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8) 2024-03-10T00:54:24.338 INFO:journalctl@ceph.osd.0.smithi106.stdout:Mar 10 00:54:24 smithi106 podman[78185]: 2024-03-10 00:54:23.956454356 +0000 UTC m=+0.018945388 image pull 01a4f6a6bb3e92ab9ab39a2f835684ea9b10e6ec9fb81ac08ebea4165583a538 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bc4a24690ce5d7255d490e21058d8c114b55e1b93a3d8345f90b6700ad29dada 2024-03-10T00:54:24.338 INFO:journalctl@ceph.osd.0.smithi106.stdout:Mar 10 00:54:24 smithi106 podman[78185]: 2024-03-10 00:54:24.061815638 +0000 UTC m=+0.124306664 container create f167944af3ea033f3b8fb7a6baa0564587aab7109c9ecda71549462c879d7e47 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bc4a24690ce5d7255d490e21058d8c114b55e1b93a3d8345f90b6700ad29dada, name=ceph-75c37498-de77-11ee-95c7-87774f69a715-osd-0-deactivate, ceph=True, GIT_BRANCH=HEAD, org.label-schema.build-date=20240222, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-yuri4-testing-2024-03-05-0854-831bc18, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_CLEAN=True) 2024-03-10T00:54:24.338 INFO:journalctl@ceph.osd.0.smithi106.stdout:Mar 10 00:54:24 smithi106 podman[78185]: 2024-03-10 00:54:24.27267081 +0000 UTC m=+0.335161849 container init f167944af3ea033f3b8fb7a6baa0564587aab7109c9ecda71549462c879d7e47 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bc4a24690ce5d7255d490e21058d8c114b55e1b93a3d8345f90b6700ad29dada, name=ceph-75c37498-de77-11ee-95c7-87774f69a715-osd-0-deactivate, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240222, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, RELEASE=wip-yuri4-testing-2024-03-05-0854-831bc18, org.label-schema.license=GPLv2, ceph=True) 2024-03-10T00:54:24.338 INFO:journalctl@ceph.osd.0.smithi106.stdout:Mar 10 00:54:24 smithi106 podman[78185]: 2024-03-10 00:54:24.278401675 +0000 UTC m=+0.340892702 container start f167944af3ea033f3b8fb7a6baa0564587aab7109c9ecda71549462c879d7e47 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bc4a24690ce5d7255d490e21058d8c114b55e1b93a3d8345f90b6700ad29dada, name=ceph-75c37498-de77-11ee-95c7-87774f69a715-osd-0-deactivate, RELEASE=wip-yuri4-testing-2024-03-05-0854-831bc18, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240222, org.label-schema.schema-version=1.0, ceph=True, GIT_BRANCH=HEAD, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-10T00:54:24.838 INFO:journalctl@ceph.osd.0.smithi106.stdout:Mar 10 00:54:24 smithi106 podman[78185]: 2024-03-10 00:54:24.372474295 +0000 UTC m=+0.434965327 container attach f167944af3ea033f3b8fb7a6baa0564587aab7109c9ecda71549462c879d7e47 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bc4a24690ce5d7255d490e21058d8c114b55e1b93a3d8345f90b6700ad29dada, name=ceph-75c37498-de77-11ee-95c7-87774f69a715-osd-0-deactivate, RELEASE=wip-yuri4-testing-2024-03-05-0854-831bc18, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240222, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-10T00:54:24.838 INFO:journalctl@ceph.osd.0.smithi106.stdout:Mar 10 00:54:24 smithi106 podman[78185]: 2024-03-10 00:54:24.570692718 +0000 UTC m=+0.633183746 container died f167944af3ea033f3b8fb7a6baa0564587aab7109c9ecda71549462c879d7e47 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bc4a24690ce5d7255d490e21058d8c114b55e1b93a3d8345f90b6700ad29dada, name=ceph-75c37498-de77-11ee-95c7-87774f69a715-osd-0-deactivate, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, RELEASE=wip-yuri4-testing-2024-03-05-0854-831bc18, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, ceph=True, org.label-schema.build-date=20240222) 2024-03-10T00:54:25.427 DEBUG:teuthology.orchestra.run.smithi106:> sudo pkill -f 'journalctl -f -n 0 -u ceph-75c37498-de77-11ee-95c7-87774f69a715@osd.0.service' 2024-03-10T00:54:25.470 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-10T00:54:25.470 INFO:tasks.cephadm.osd.0:Stopped osd.0 2024-03-10T00:54:25.470 INFO:tasks.cephadm.osd.1:Stopping osd.1... 2024-03-10T00:54:25.470 DEBUG:teuthology.orchestra.run.smithi106:> sudo systemctl stop ceph-75c37498-de77-11ee-95c7-87774f69a715@osd.1 2024-03-10T00:54:25.804 INFO:journalctl@ceph.osd.1.smithi106.stdout:Mar 10 00:54:25 smithi106 systemd[1]: Stopping Ceph osd.1 for 75c37498-de77-11ee-95c7-87774f69a715... 2024-03-10T00:54:26.088 INFO:journalctl@ceph.osd.1.smithi106.stdout:Mar 10 00:54:25 smithi106 ceph-75c37498-de77-11ee-95c7-87774f69a715-osd-1[42365]: 2024-03-10T00:54:25.806+0000 7f3c2f70f700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.1 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-10T00:54:26.088 INFO:journalctl@ceph.osd.1.smithi106.stdout:Mar 10 00:54:25 smithi106 ceph-75c37498-de77-11ee-95c7-87774f69a715-osd-1[42365]: 2024-03-10T00:54:25.806+0000 7f3c2f70f700 -1 osd.1 71 *** Got signal Terminated *** 2024-03-10T00:54:26.088 INFO:journalctl@ceph.osd.1.smithi106.stdout:Mar 10 00:54:25 smithi106 ceph-75c37498-de77-11ee-95c7-87774f69a715-osd-1[42365]: 2024-03-10T00:54:25.806+0000 7f3c2f70f700 -1 osd.1 71 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-10T00:54:29.374 INFO:journalctl@ceph.prometheus.a.smithi170.stdout:Mar 10 00:54:29 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-prometheus-a[52583]: ts=2024-03-10T00:54:29.079Z caller=refresh.go:99 level=error component="discovery manager scrape" discovery=http config=ceph-exporter msg="Unable to refresh target groups" err="Get \"http://172.21.15.106:8765/sd/prometheus/sd-config?service=ceph-exporter\": dial tcp 172.21.15.106:8765: connect: connection refused" 2024-03-10T00:54:29.374 INFO:journalctl@ceph.prometheus.a.smithi170.stdout:Mar 10 00:54:29 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-prometheus-a[52583]: ts=2024-03-10T00:54:29.079Z caller=refresh.go:99 level=error component="discovery manager scrape" discovery=http config=ceph msg="Unable to refresh target groups" err="Get \"http://172.21.15.106:8765/sd/prometheus/sd-config?service=mgr-prometheus\": dial tcp 172.21.15.106:8765: connect: connection refused" 2024-03-10T00:54:29.374 INFO:journalctl@ceph.prometheus.a.smithi170.stdout:Mar 10 00:54:29 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-prometheus-a[52583]: ts=2024-03-10T00:54:29.079Z caller=refresh.go:99 level=error component="discovery manager scrape" discovery=http config=nvmeof msg="Unable to refresh target groups" err="Get \"http://172.21.15.106:8765/sd/prometheus/sd-config?service=nvmeof\": dial tcp 172.21.15.106:8765: connect: connection refused" 2024-03-10T00:54:31.588 INFO:journalctl@ceph.osd.1.smithi106.stdout:Mar 10 00:54:31 smithi106 podman[78316]: 2024-03-10 00:54:31.265392988 +0000 UTC m=+5.568371574 container died 2a4d6d60877d3b9d0cdaa091798522bfbeb3e0a458949c98ac2c23c1bf4821e2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bc4a24690ce5d7255d490e21058d8c114b55e1b93a3d8345f90b6700ad29dada, name=ceph-75c37498-de77-11ee-95c7-87774f69a715-osd-1, org.label-schema.vendor=CentOS, RELEASE=wip-yuri4-testing-2024-03-05-0854-831bc18, ceph=True, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240222, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5) 2024-03-10T00:54:31.910 INFO:journalctl@ceph.osd.1.smithi106.stdout:Mar 10 00:54:31 smithi106 podman[78316]: 2024-03-10 00:54:31.678540575 +0000 UTC m=+5.981519157 container cleanup 2a4d6d60877d3b9d0cdaa091798522bfbeb3e0a458949c98ac2c23c1bf4821e2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bc4a24690ce5d7255d490e21058d8c114b55e1b93a3d8345f90b6700ad29dada, name=ceph-75c37498-de77-11ee-95c7-87774f69a715-osd-1, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240222, RELEASE=wip-yuri4-testing-2024-03-05-0854-831bc18, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, ceph=True, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS) 2024-03-10T00:54:31.910 INFO:journalctl@ceph.osd.1.smithi106.stdout:Mar 10 00:54:31 smithi106 bash[78316]: ceph-75c37498-de77-11ee-95c7-87774f69a715-osd-1 2024-03-10T00:54:32.337 INFO:journalctl@ceph.osd.1.smithi106.stdout:Mar 10 00:54:31 smithi106 podman[78568]: 2024-03-10 00:54:31.913034058 +0000 UTC m=+0.779561945 container remove 2a4d6d60877d3b9d0cdaa091798522bfbeb3e0a458949c98ac2c23c1bf4821e2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bc4a24690ce5d7255d490e21058d8c114b55e1b93a3d8345f90b6700ad29dada, name=ceph-75c37498-de77-11ee-95c7-87774f69a715-osd-1, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240222, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , RELEASE=wip-yuri4-testing-2024-03-05-0854-831bc18, GIT_CLEAN=True, ceph=True, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-10T00:54:32.702 INFO:journalctl@ceph.osd.1.smithi106.stdout:Mar 10 00:54:32 smithi106 podman[78738]: 2024-03-10 00:54:32.478676571 +0000 UTC m=+0.019499496 image pull 01a4f6a6bb3e92ab9ab39a2f835684ea9b10e6ec9fb81ac08ebea4165583a538 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bc4a24690ce5d7255d490e21058d8c114b55e1b93a3d8345f90b6700ad29dada 2024-03-10T00:54:32.702 INFO:journalctl@ceph.osd.1.smithi106.stdout:Mar 10 00:54:32 smithi106 podman[78738]: 2024-03-10 00:54:32.59812766 +0000 UTC m=+0.138950627 container create 474571778b3f13257aabb765d60ca7524c380ae2dd6d717273eef98006f7b934 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bc4a24690ce5d7255d490e21058d8c114b55e1b93a3d8345f90b6700ad29dada, name=ceph-75c37498-de77-11ee-95c7-87774f69a715-osd-1-deactivate, maintainer=Guillaume Abrioux , ceph=True, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, RELEASE=wip-yuri4-testing-2024-03-05-0854-831bc18, org.label-schema.build-date=20240222, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_CLEAN=True) 2024-03-10T00:54:32.956 INFO:journalctl@ceph.osd.1.smithi106.stdout:Mar 10 00:54:32 smithi106 podman[78738]: 2024-03-10 00:54:32.850690999 +0000 UTC m=+0.391513925 container init 474571778b3f13257aabb765d60ca7524c380ae2dd6d717273eef98006f7b934 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bc4a24690ce5d7255d490e21058d8c114b55e1b93a3d8345f90b6700ad29dada, name=ceph-75c37498-de77-11ee-95c7-87774f69a715-osd-1-deactivate, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, ceph=True, RELEASE=wip-yuri4-testing-2024-03-05-0854-831bc18, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.build-date=20240222) 2024-03-10T00:54:32.956 INFO:journalctl@ceph.osd.1.smithi106.stdout:Mar 10 00:54:32 smithi106 podman[78738]: 2024-03-10 00:54:32.856169953 +0000 UTC m=+0.396992875 container start 474571778b3f13257aabb765d60ca7524c380ae2dd6d717273eef98006f7b934 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bc4a24690ce5d7255d490e21058d8c114b55e1b93a3d8345f90b6700ad29dada, name=ceph-75c37498-de77-11ee-95c7-87774f69a715-osd-1-deactivate, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, RELEASE=wip-yuri4-testing-2024-03-05-0854-831bc18, maintainer=Guillaume Abrioux , ceph=True, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240222, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8) 2024-03-10T00:54:32.956 INFO:journalctl@ceph.osd.1.smithi106.stdout:Mar 10 00:54:32 smithi106 podman[78738]: 2024-03-10 00:54:32.959182869 +0000 UTC m=+0.500005787 container attach 474571778b3f13257aabb765d60ca7524c380ae2dd6d717273eef98006f7b934 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bc4a24690ce5d7255d490e21058d8c114b55e1b93a3d8345f90b6700ad29dada, name=ceph-75c37498-de77-11ee-95c7-87774f69a715-osd-1-deactivate, GIT_CLEAN=True, RELEASE=wip-yuri4-testing-2024-03-05-0854-831bc18, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240222) 2024-03-10T00:54:33.276 INFO:journalctl@ceph.osd.1.smithi106.stdout:Mar 10 00:54:33 smithi106 podman[78738]: 2024-03-10 00:54:33.177466696 +0000 UTC m=+0.718289614 container died 474571778b3f13257aabb765d60ca7524c380ae2dd6d717273eef98006f7b934 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bc4a24690ce5d7255d490e21058d8c114b55e1b93a3d8345f90b6700ad29dada, name=ceph-75c37498-de77-11ee-95c7-87774f69a715-osd-1-deactivate, RELEASE=wip-yuri4-testing-2024-03-05-0854-831bc18, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240222, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-10T00:54:34.318 DEBUG:teuthology.orchestra.run.smithi106:> sudo pkill -f 'journalctl -f -n 0 -u ceph-75c37498-de77-11ee-95c7-87774f69a715@osd.1.service' 2024-03-10T00:54:34.369 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-10T00:54:34.369 INFO:tasks.cephadm.osd.1:Stopped osd.1 2024-03-10T00:54:34.370 INFO:tasks.cephadm.osd.2:Stopping osd.2... 2024-03-10T00:54:34.370 DEBUG:teuthology.orchestra.run.smithi106:> sudo systemctl stop ceph-75c37498-de77-11ee-95c7-87774f69a715@osd.2 2024-03-10T00:54:34.588 INFO:journalctl@ceph.osd.2.smithi106.stdout:Mar 10 00:54:34 smithi106 systemd[1]: Stopping Ceph osd.2 for 75c37498-de77-11ee-95c7-87774f69a715... 2024-03-10T00:54:35.088 INFO:journalctl@ceph.osd.2.smithi106.stdout:Mar 10 00:54:34 smithi106 ceph-75c37498-de77-11ee-95c7-87774f69a715-osd-2[46309]: 2024-03-10T00:54:34.693+0000 7f70aa38d700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.2 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-10T00:54:35.088 INFO:journalctl@ceph.osd.2.smithi106.stdout:Mar 10 00:54:34 smithi106 ceph-75c37498-de77-11ee-95c7-87774f69a715-osd-2[46309]: 2024-03-10T00:54:34.693+0000 7f70aa38d700 -1 osd.2 71 *** Got signal Terminated *** 2024-03-10T00:54:35.088 INFO:journalctl@ceph.osd.2.smithi106.stdout:Mar 10 00:54:34 smithi106 ceph-75c37498-de77-11ee-95c7-87774f69a715-osd-2[46309]: 2024-03-10T00:54:34.693+0000 7f70aa38d700 -1 osd.2 71 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-10T00:54:40.416 INFO:journalctl@ceph.osd.2.smithi106.stdout:Mar 10 00:54:40 smithi106 podman[78977]: 2024-03-10 00:54:40.123758394 +0000 UTC m=+5.544162505 container died 487a0b16efc77124dad3a0abc9adead19803f6fe297da1a829eac1d0d5120cbb (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bc4a24690ce5d7255d490e21058d8c114b55e1b93a3d8345f90b6700ad29dada, name=ceph-75c37498-de77-11ee-95c7-87774f69a715-osd-2, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.schema-version=1.0, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, GIT_CLEAN=True, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240222, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, RELEASE=wip-yuri4-testing-2024-03-05-0854-831bc18, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-10T00:54:40.838 INFO:journalctl@ceph.osd.2.smithi106.stdout:Mar 10 00:54:40 smithi106 podman[78977]: 2024-03-10 00:54:40.591356014 +0000 UTC m=+6.011760119 container cleanup 487a0b16efc77124dad3a0abc9adead19803f6fe297da1a829eac1d0d5120cbb (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bc4a24690ce5d7255d490e21058d8c114b55e1b93a3d8345f90b6700ad29dada, name=ceph-75c37498-de77-11ee-95c7-87774f69a715-osd-2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240222, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_CLEAN=True, ceph=True, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, RELEASE=wip-yuri4-testing-2024-03-05-0854-831bc18, io.buildah.version=1.33.5) 2024-03-10T00:54:40.838 INFO:journalctl@ceph.osd.2.smithi106.stdout:Mar 10 00:54:40 smithi106 bash[78977]: ceph-75c37498-de77-11ee-95c7-87774f69a715-osd-2 2024-03-10T00:54:41.140 INFO:journalctl@ceph.osd.2.smithi106.stdout:Mar 10 00:54:40 smithi106 podman[79388]: 2024-03-10 00:54:40.842490655 +0000 UTC m=+0.712303192 container remove 487a0b16efc77124dad3a0abc9adead19803f6fe297da1a829eac1d0d5120cbb (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bc4a24690ce5d7255d490e21058d8c114b55e1b93a3d8345f90b6700ad29dada, name=ceph-75c37498-de77-11ee-95c7-87774f69a715-osd-2, org.label-schema.build-date=20240222, RELEASE=wip-yuri4-testing-2024-03-05-0854-831bc18, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, org.label-schema.vendor=CentOS, ceph=True, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, io.buildah.version=1.33.5) 2024-03-10T00:54:41.442 INFO:journalctl@ceph.osd.2.smithi106.stdout:Mar 10 00:54:41 smithi106 podman[79483]: 2024-03-10 00:54:41.143259241 +0000 UTC m=+0.109292159 container create d3314a7f4bce41fcb14c3caf1f11e78ab7a1f3b43da19eafb16c23ad276a7616 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bc4a24690ce5d7255d490e21058d8c114b55e1b93a3d8345f90b6700ad29dada, name=ceph-75c37498-de77-11ee-95c7-87774f69a715-osd-2-deactivate, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, ceph=True, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-yuri4-testing-2024-03-05-0854-831bc18, org.label-schema.build-date=20240222, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-10T00:54:41.443 INFO:journalctl@ceph.osd.2.smithi106.stdout:Mar 10 00:54:41 smithi106 podman[79483]: 2024-03-10 00:54:41.051546666 +0000 UTC m=+0.017579599 image pull 01a4f6a6bb3e92ab9ab39a2f835684ea9b10e6ec9fb81ac08ebea4165583a538 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bc4a24690ce5d7255d490e21058d8c114b55e1b93a3d8345f90b6700ad29dada 2024-03-10T00:54:41.443 INFO:journalctl@ceph.osd.2.smithi106.stdout:Mar 10 00:54:41 smithi106 podman[79483]: 2024-03-10 00:54:41.353905479 +0000 UTC m=+0.319938399 container init d3314a7f4bce41fcb14c3caf1f11e78ab7a1f3b43da19eafb16c23ad276a7616 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bc4a24690ce5d7255d490e21058d8c114b55e1b93a3d8345f90b6700ad29dada, name=ceph-75c37498-de77-11ee-95c7-87774f69a715-osd-2-deactivate, GIT_CLEAN=True, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, ceph=True, org.label-schema.build-date=20240222, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, RELEASE=wip-yuri4-testing-2024-03-05-0854-831bc18, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-10T00:54:41.443 INFO:journalctl@ceph.osd.2.smithi106.stdout:Mar 10 00:54:41 smithi106 podman[79483]: 2024-03-10 00:54:41.359818902 +0000 UTC m=+0.325851825 container start d3314a7f4bce41fcb14c3caf1f11e78ab7a1f3b43da19eafb16c23ad276a7616 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bc4a24690ce5d7255d490e21058d8c114b55e1b93a3d8345f90b6700ad29dada, name=ceph-75c37498-de77-11ee-95c7-87774f69a715-osd-2-deactivate, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240222, GIT_BRANCH=HEAD, ceph=True, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.vendor=CentOS, RELEASE=wip-yuri4-testing-2024-03-05-0854-831bc18) 2024-03-10T00:54:41.443 INFO:journalctl@ceph.osd.2.smithi106.stdout:Mar 10 00:54:41 smithi106 podman[79483]: 2024-03-10 00:54:41.445390292 +0000 UTC m=+0.411423230 container attach d3314a7f4bce41fcb14c3caf1f11e78ab7a1f3b43da19eafb16c23ad276a7616 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bc4a24690ce5d7255d490e21058d8c114b55e1b93a3d8345f90b6700ad29dada, name=ceph-75c37498-de77-11ee-95c7-87774f69a715-osd-2-deactivate, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240222, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , RELEASE=wip-yuri4-testing-2024-03-05-0854-831bc18) 2024-03-10T00:54:41.838 INFO:journalctl@ceph.osd.2.smithi106.stdout:Mar 10 00:54:41 smithi106 podman[79483]: 2024-03-10 00:54:41.653219083 +0000 UTC m=+0.619252003 container died d3314a7f4bce41fcb14c3caf1f11e78ab7a1f3b43da19eafb16c23ad276a7616 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bc4a24690ce5d7255d490e21058d8c114b55e1b93a3d8345f90b6700ad29dada, name=ceph-75c37498-de77-11ee-95c7-87774f69a715-osd-2-deactivate, org.label-schema.build-date=20240222, io.buildah.version=1.33.5, RELEASE=wip-yuri4-testing-2024-03-05-0854-831bc18, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, ceph=True, GIT_BRANCH=HEAD, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8) 2024-03-10T00:54:42.457 DEBUG:teuthology.orchestra.run.smithi106:> sudo pkill -f 'journalctl -f -n 0 -u ceph-75c37498-de77-11ee-95c7-87774f69a715@osd.2.service' 2024-03-10T00:54:42.501 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-10T00:54:42.501 INFO:tasks.cephadm.osd.2:Stopped osd.2 2024-03-10T00:54:42.501 INFO:tasks.cephadm.osd.3:Stopping osd.3... 2024-03-10T00:54:42.502 DEBUG:teuthology.orchestra.run.smithi106:> sudo systemctl stop ceph-75c37498-de77-11ee-95c7-87774f69a715@osd.3 2024-03-10T00:54:42.819 INFO:journalctl@ceph.osd.3.smithi106.stdout:Mar 10 00:54:42 smithi106 systemd[1]: Stopping Ceph osd.3 for 75c37498-de77-11ee-95c7-87774f69a715... 2024-03-10T00:54:43.087 INFO:journalctl@ceph.osd.3.smithi106.stdout:Mar 10 00:54:42 smithi106 ceph-75c37498-de77-11ee-95c7-87774f69a715-osd-3[50617]: 2024-03-10T00:54:42.821+0000 7f80a1517700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.3 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-10T00:54:43.088 INFO:journalctl@ceph.osd.3.smithi106.stdout:Mar 10 00:54:42 smithi106 ceph-75c37498-de77-11ee-95c7-87774f69a715-osd-3[50617]: 2024-03-10T00:54:42.821+0000 7f80a1517700 -1 osd.3 71 *** Got signal Terminated *** 2024-03-10T00:54:43.088 INFO:journalctl@ceph.osd.3.smithi106.stdout:Mar 10 00:54:42 smithi106 ceph-75c37498-de77-11ee-95c7-87774f69a715-osd-3[50617]: 2024-03-10T00:54:42.821+0000 7f80a1517700 -1 osd.3 71 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-10T00:54:44.338 INFO:journalctl@ceph.osd.3.smithi106.stdout:Mar 10 00:54:43 smithi106 ceph-75c37498-de77-11ee-95c7-87774f69a715-osd-3[50617]: 2024-03-10T00:54:43.904+0000 7f8099b03700 -1 osd.3 71 heartbeat_check: no reply from 172.21.15.106:6806 osd.0 since back 2024-03-10T00:54:18.390476+0000 front 2024-03-10T00:54:18.390403+0000 (oldest deadline 2024-03-10T00:54:43.090611+0000) 2024-03-10T00:54:44.774 INFO:journalctl@ceph.osd.6.smithi170.stdout:Mar 10 00:54:44 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-osd-6[43049]: 2024-03-10T00:54:44.431+0000 7f5852423700 -1 osd.6 71 heartbeat_check: no reply from 172.21.15.106:6806 osd.0 since back 2024-03-10T00:54:17.852283+0000 front 2024-03-10T00:54:17.852086+0000 (oldest deadline 2024-03-10T00:54:43.752263+0000) 2024-03-10T00:54:44.774 INFO:journalctl@ceph.osd.7.smithi170.stdout:Mar 10 00:54:44 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-osd-7[47108]: 2024-03-10T00:54:44.344+0000 7f7b81779700 -1 osd.7 71 heartbeat_check: no reply from 172.21.15.106:6806 osd.0 since back 2024-03-10T00:54:19.432267+0000 front 2024-03-10T00:54:19.432106+0000 (oldest deadline 2024-03-10T00:54:43.532104+0000) 2024-03-10T00:54:45.338 INFO:journalctl@ceph.osd.3.smithi106.stdout:Mar 10 00:54:44 smithi106 ceph-75c37498-de77-11ee-95c7-87774f69a715-osd-3[50617]: 2024-03-10T00:54:44.927+0000 7f8099b03700 -1 osd.3 71 heartbeat_check: no reply from 172.21.15.106:6806 osd.0 since back 2024-03-10T00:54:18.390476+0000 front 2024-03-10T00:54:18.390403+0000 (oldest deadline 2024-03-10T00:54:43.090611+0000) 2024-03-10T00:54:45.524 INFO:journalctl@ceph.osd.6.smithi170.stdout:Mar 10 00:54:45 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-osd-6[43049]: 2024-03-10T00:54:45.397+0000 7f5852423700 -1 osd.6 71 heartbeat_check: no reply from 172.21.15.106:6806 osd.0 since back 2024-03-10T00:54:17.852283+0000 front 2024-03-10T00:54:17.852086+0000 (oldest deadline 2024-03-10T00:54:43.752263+0000) 2024-03-10T00:54:45.524 INFO:journalctl@ceph.osd.7.smithi170.stdout:Mar 10 00:54:45 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-osd-7[47108]: 2024-03-10T00:54:45.345+0000 7f7b81779700 -1 osd.7 71 heartbeat_check: no reply from 172.21.15.106:6806 osd.0 since back 2024-03-10T00:54:19.432267+0000 front 2024-03-10T00:54:19.432106+0000 (oldest deadline 2024-03-10T00:54:43.532104+0000) 2024-03-10T00:54:45.963 INFO:journalctl@ceph.osd.4.smithi170.stdout:Mar 10 00:54:45 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-osd-4[35111]: 2024-03-10T00:54:45.619+0000 7f053cb3d700 -1 osd.4 71 heartbeat_check: no reply from 172.21.15.106:6806 osd.0 since back 2024-03-10T00:54:19.067553+0000 front 2024-03-10T00:54:19.067498+0000 (oldest deadline 2024-03-10T00:54:44.967307+0000) 2024-03-10T00:54:46.239 INFO:journalctl@ceph.osd.3.smithi106.stdout:Mar 10 00:54:45 smithi106 ceph-75c37498-de77-11ee-95c7-87774f69a715-osd-3[50617]: 2024-03-10T00:54:45.972+0000 7f8099b03700 -1 osd.3 71 heartbeat_check: no reply from 172.21.15.106:6806 osd.0 since back 2024-03-10T00:54:18.390476+0000 front 2024-03-10T00:54:18.390403+0000 (oldest deadline 2024-03-10T00:54:43.090611+0000) 2024-03-10T00:54:46.774 INFO:journalctl@ceph.osd.7.smithi170.stdout:Mar 10 00:54:46 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-osd-7[47108]: 2024-03-10T00:54:46.325+0000 7f7b81779700 -1 osd.7 71 heartbeat_check: no reply from 172.21.15.106:6806 osd.0 since back 2024-03-10T00:54:19.432267+0000 front 2024-03-10T00:54:19.432106+0000 (oldest deadline 2024-03-10T00:54:43.532104+0000) 2024-03-10T00:54:46.775 INFO:journalctl@ceph.osd.4.smithi170.stdout:Mar 10 00:54:46 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-osd-4[35111]: 2024-03-10T00:54:46.574+0000 7f053cb3d700 -1 osd.4 71 heartbeat_check: no reply from 172.21.15.106:6806 osd.0 since back 2024-03-10T00:54:19.067553+0000 front 2024-03-10T00:54:19.067498+0000 (oldest deadline 2024-03-10T00:54:44.967307+0000) 2024-03-10T00:54:46.775 INFO:journalctl@ceph.osd.6.smithi170.stdout:Mar 10 00:54:46 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-osd-6[43049]: 2024-03-10T00:54:46.351+0000 7f5852423700 -1 osd.6 71 heartbeat_check: no reply from 172.21.15.106:6806 osd.0 since back 2024-03-10T00:54:17.852283+0000 front 2024-03-10T00:54:17.852086+0000 (oldest deadline 2024-03-10T00:54:43.752263+0000) 2024-03-10T00:54:47.338 INFO:journalctl@ceph.osd.3.smithi106.stdout:Mar 10 00:54:47 smithi106 ceph-75c37498-de77-11ee-95c7-87774f69a715-osd-3[50617]: 2024-03-10T00:54:47.021+0000 7f8099b03700 -1 osd.3 71 heartbeat_check: no reply from 172.21.15.106:6806 osd.0 since back 2024-03-10T00:54:18.390476+0000 front 2024-03-10T00:54:18.390403+0000 (oldest deadline 2024-03-10T00:54:43.090611+0000) 2024-03-10T00:54:47.571 INFO:journalctl@ceph.osd.6.smithi170.stdout:Mar 10 00:54:47 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-osd-6[43049]: 2024-03-10T00:54:47.352+0000 7f5852423700 -1 osd.6 71 heartbeat_check: no reply from 172.21.15.106:6806 osd.0 since back 2024-03-10T00:54:17.852283+0000 front 2024-03-10T00:54:17.852086+0000 (oldest deadline 2024-03-10T00:54:43.752263+0000) 2024-03-10T00:54:47.572 INFO:journalctl@ceph.osd.7.smithi170.stdout:Mar 10 00:54:47 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-osd-7[47108]: 2024-03-10T00:54:47.290+0000 7f7b81779700 -1 osd.7 71 heartbeat_check: no reply from 172.21.15.106:6806 osd.0 since back 2024-03-10T00:54:19.432267+0000 front 2024-03-10T00:54:19.432106+0000 (oldest deadline 2024-03-10T00:54:43.532104+0000) 2024-03-10T00:54:48.024 INFO:journalctl@ceph.osd.4.smithi170.stdout:Mar 10 00:54:47 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-osd-4[35111]: 2024-03-10T00:54:47.573+0000 7f053cb3d700 -1 osd.4 71 heartbeat_check: no reply from 172.21.15.106:6806 osd.0 since back 2024-03-10T00:54:19.067553+0000 front 2024-03-10T00:54:19.067498+0000 (oldest deadline 2024-03-10T00:54:44.967307+0000) 2024-03-10T00:54:48.524 INFO:journalctl@ceph.osd.6.smithi170.stdout:Mar 10 00:54:48 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-osd-6[43049]: 2024-03-10T00:54:48.351+0000 7f5852423700 -1 osd.6 71 heartbeat_check: no reply from 172.21.15.106:6806 osd.0 since back 2024-03-10T00:54:17.852283+0000 front 2024-03-10T00:54:17.852086+0000 (oldest deadline 2024-03-10T00:54:43.752263+0000) 2024-03-10T00:54:48.524 INFO:journalctl@ceph.osd.5.smithi170.stdout:Mar 10 00:54:48 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-osd-5[38986]: 2024-03-10T00:54:48.399+0000 7f72444e9700 -1 osd.5 71 heartbeat_check: no reply from 172.21.15.106:6806 osd.0 since back 2024-03-10T00:54:18.209644+0000 front 2024-03-10T00:54:18.209502+0000 (oldest deadline 2024-03-10T00:54:48.210011+0000) 2024-03-10T00:54:48.524 INFO:journalctl@ceph.osd.7.smithi170.stdout:Mar 10 00:54:48 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-osd-7[47108]: 2024-03-10T00:54:48.249+0000 7f7b81779700 -1 osd.7 71 heartbeat_check: no reply from 172.21.15.106:6806 osd.0 since back 2024-03-10T00:54:19.432267+0000 front 2024-03-10T00:54:19.432106+0000 (oldest deadline 2024-03-10T00:54:43.532104+0000) 2024-03-10T00:54:48.691 INFO:journalctl@ceph.osd.3.smithi106.stdout:Mar 10 00:54:48 smithi106 podman[79612]: 2024-03-10 00:54:48.372442468 +0000 UTC m=+5.645821309 container died eb35f2f39ae87b5142874b2077885691cc1157c4bf8d18bab6d50519a27362e8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bc4a24690ce5d7255d490e21058d8c114b55e1b93a3d8345f90b6700ad29dada, name=ceph-75c37498-de77-11ee-95c7-87774f69a715-osd-3, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240222, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, ceph=True, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , RELEASE=wip-yuri4-testing-2024-03-05-0854-831bc18, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0) 2024-03-10T00:54:49.024 INFO:journalctl@ceph.osd.4.smithi170.stdout:Mar 10 00:54:48 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-osd-4[35111]: 2024-03-10T00:54:48.580+0000 7f053cb3d700 -1 osd.4 71 heartbeat_check: no reply from 172.21.15.106:6806 osd.0 since back 2024-03-10T00:54:19.067553+0000 front 2024-03-10T00:54:19.067498+0000 (oldest deadline 2024-03-10T00:54:44.967307+0000) 2024-03-10T00:54:49.088 INFO:journalctl@ceph.osd.3.smithi106.stdout:Mar 10 00:54:48 smithi106 podman[79612]: 2024-03-10 00:54:48.876455321 +0000 UTC m=+6.149834168 container cleanup eb35f2f39ae87b5142874b2077885691cc1157c4bf8d18bab6d50519a27362e8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bc4a24690ce5d7255d490e21058d8c114b55e1b93a3d8345f90b6700ad29dada, name=ceph-75c37498-de77-11ee-95c7-87774f69a715-osd-3, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240222, RELEASE=wip-yuri4-testing-2024-03-05-0854-831bc18, ceph=True) 2024-03-10T00:54:49.088 INFO:journalctl@ceph.osd.3.smithi106.stdout:Mar 10 00:54:48 smithi106 bash[79612]: ceph-75c37498-de77-11ee-95c7-87774f69a715-osd-3 2024-03-10T00:54:49.482 INFO:journalctl@ceph.osd.3.smithi106.stdout:Mar 10 00:54:49 smithi106 podman[79645]: 2024-03-10 00:54:49.152710904 +0000 UTC m=+0.770981161 container remove eb35f2f39ae87b5142874b2077885691cc1157c4bf8d18bab6d50519a27362e8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bc4a24690ce5d7255d490e21058d8c114b55e1b93a3d8345f90b6700ad29dada, name=ceph-75c37498-de77-11ee-95c7-87774f69a715-osd-3, RELEASE=wip-yuri4-testing-2024-03-05-0854-831bc18, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, ceph=True, org.label-schema.build-date=20240222, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5) 2024-03-10T00:54:49.524 INFO:journalctl@ceph.osd.5.smithi170.stdout:Mar 10 00:54:49 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-osd-5[38986]: 2024-03-10T00:54:49.423+0000 7f72444e9700 -1 osd.5 71 heartbeat_check: no reply from 172.21.15.106:6806 osd.0 since back 2024-03-10T00:54:18.209644+0000 front 2024-03-10T00:54:18.209502+0000 (oldest deadline 2024-03-10T00:54:48.210011+0000) 2024-03-10T00:54:49.524 INFO:journalctl@ceph.osd.7.smithi170.stdout:Mar 10 00:54:49 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-osd-7[47108]: 2024-03-10T00:54:49.217+0000 7f7b81779700 -1 osd.7 71 heartbeat_check: no reply from 172.21.15.106:6806 osd.0 since back 2024-03-10T00:54:19.432267+0000 front 2024-03-10T00:54:19.432106+0000 (oldest deadline 2024-03-10T00:54:43.532104+0000) 2024-03-10T00:54:49.524 INFO:journalctl@ceph.osd.6.smithi170.stdout:Mar 10 00:54:49 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-osd-6[43049]: 2024-03-10T00:54:49.316+0000 7f5852423700 -1 osd.6 71 heartbeat_check: no reply from 172.21.15.106:6806 osd.0 since back 2024-03-10T00:54:17.852283+0000 front 2024-03-10T00:54:17.852086+0000 (oldest deadline 2024-03-10T00:54:43.752263+0000) 2024-03-10T00:54:49.794 INFO:journalctl@ceph.osd.3.smithi106.stdout:Mar 10 00:54:49 smithi106 podman[79745]: 2024-03-10 00:54:49.388218423 +0000 UTC m=+0.017866895 image pull 01a4f6a6bb3e92ab9ab39a2f835684ea9b10e6ec9fb81ac08ebea4165583a538 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bc4a24690ce5d7255d490e21058d8c114b55e1b93a3d8345f90b6700ad29dada 2024-03-10T00:54:49.794 INFO:journalctl@ceph.osd.3.smithi106.stdout:Mar 10 00:54:49 smithi106 podman[79745]: 2024-03-10 00:54:49.486626486 +0000 UTC m=+0.116274958 container create 42eaace2f85f70ecc7ccc336593620f285f4a238a5275bcc6b3e6039050ab1f3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bc4a24690ce5d7255d490e21058d8c114b55e1b93a3d8345f90b6700ad29dada, name=ceph-75c37498-de77-11ee-95c7-87774f69a715-osd-3-deactivate, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, RELEASE=wip-yuri4-testing-2024-03-05-0854-831bc18, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240222, GIT_CLEAN=True) 2024-03-10T00:54:49.794 INFO:journalctl@ceph.osd.3.smithi106.stdout:Mar 10 00:54:49 smithi106 podman[79745]: 2024-03-10 00:54:49.705808374 +0000 UTC m=+0.335456847 container init 42eaace2f85f70ecc7ccc336593620f285f4a238a5275bcc6b3e6039050ab1f3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bc4a24690ce5d7255d490e21058d8c114b55e1b93a3d8345f90b6700ad29dada, name=ceph-75c37498-de77-11ee-95c7-87774f69a715-osd-3-deactivate, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240222, RELEASE=wip-yuri4-testing-2024-03-05-0854-831bc18, maintainer=Guillaume Abrioux ) 2024-03-10T00:54:49.794 INFO:journalctl@ceph.osd.3.smithi106.stdout:Mar 10 00:54:49 smithi106 podman[79745]: 2024-03-10 00:54:49.7120577 +0000 UTC m=+0.341706173 container start 42eaace2f85f70ecc7ccc336593620f285f4a238a5275bcc6b3e6039050ab1f3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bc4a24690ce5d7255d490e21058d8c114b55e1b93a3d8345f90b6700ad29dada, name=ceph-75c37498-de77-11ee-95c7-87774f69a715-osd-3-deactivate, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.build-date=20240222, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, RELEASE=wip-yuri4-testing-2024-03-05-0854-831bc18) 2024-03-10T00:54:49.795 INFO:journalctl@ceph.osd.3.smithi106.stdout:Mar 10 00:54:49 smithi106 podman[79745]: 2024-03-10 00:54:49.797265185 +0000 UTC m=+0.426913666 container attach 42eaace2f85f70ecc7ccc336593620f285f4a238a5275bcc6b3e6039050ab1f3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bc4a24690ce5d7255d490e21058d8c114b55e1b93a3d8345f90b6700ad29dada, name=ceph-75c37498-de77-11ee-95c7-87774f69a715-osd-3-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.build-date=20240222, io.buildah.version=1.33.5, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, ceph=True, RELEASE=wip-yuri4-testing-2024-03-05-0854-831bc18, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=) 2024-03-10T00:54:50.024 INFO:journalctl@ceph.osd.4.smithi170.stdout:Mar 10 00:54:49 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-osd-4[35111]: 2024-03-10T00:54:49.536+0000 7f053cb3d700 -1 osd.4 71 heartbeat_check: no reply from 172.21.15.106:6806 osd.0 since back 2024-03-10T00:54:19.067553+0000 front 2024-03-10T00:54:19.067498+0000 (oldest deadline 2024-03-10T00:54:44.967307+0000) 2024-03-10T00:54:50.048 INFO:journalctl@ceph.osd.3.smithi106.stdout:Mar 10 00:54:50 smithi106 podman[79745]: 2024-03-10 00:54:50.025469721 +0000 UTC m=+0.655118206 container died 42eaace2f85f70ecc7ccc336593620f285f4a238a5275bcc6b3e6039050ab1f3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bc4a24690ce5d7255d490e21058d8c114b55e1b93a3d8345f90b6700ad29dada, name=ceph-75c37498-de77-11ee-95c7-87774f69a715-osd-3-deactivate, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240222, RELEASE=wip-yuri4-testing-2024-03-05-0854-831bc18, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, io.buildah.version=1.33.5, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8) 2024-03-10T00:54:50.524 INFO:journalctl@ceph.osd.6.smithi170.stdout:Mar 10 00:54:50 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-osd-6[43049]: 2024-03-10T00:54:50.338+0000 7f5852423700 -1 osd.6 71 heartbeat_check: no reply from 172.21.15.106:6806 osd.0 since back 2024-03-10T00:54:17.852283+0000 front 2024-03-10T00:54:17.852086+0000 (oldest deadline 2024-03-10T00:54:43.752263+0000) 2024-03-10T00:54:50.524 INFO:journalctl@ceph.osd.7.smithi170.stdout:Mar 10 00:54:50 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-osd-7[47108]: 2024-03-10T00:54:50.255+0000 7f7b81779700 -1 osd.7 71 heartbeat_check: no reply from 172.21.15.106:6806 osd.0 since back 2024-03-10T00:54:19.432267+0000 front 2024-03-10T00:54:19.432106+0000 (oldest deadline 2024-03-10T00:54:43.532104+0000) 2024-03-10T00:54:50.524 INFO:journalctl@ceph.osd.5.smithi170.stdout:Mar 10 00:54:50 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-osd-5[38986]: 2024-03-10T00:54:50.462+0000 7f72444e9700 -1 osd.5 71 heartbeat_check: no reply from 172.21.15.106:6806 osd.0 since back 2024-03-10T00:54:18.209644+0000 front 2024-03-10T00:54:18.209502+0000 (oldest deadline 2024-03-10T00:54:48.210011+0000) 2024-03-10T00:54:50.822 INFO:journalctl@ceph.osd.4.smithi170.stdout:Mar 10 00:54:50 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-osd-4[35111]: 2024-03-10T00:54:50.541+0000 7f053cb3d700 -1 osd.4 71 heartbeat_check: no reply from 172.21.15.106:6806 osd.0 since back 2024-03-10T00:54:19.067553+0000 front 2024-03-10T00:54:19.067498+0000 (oldest deadline 2024-03-10T00:54:44.967307+0000) 2024-03-10T00:54:51.192 DEBUG:teuthology.orchestra.run.smithi106:> sudo pkill -f 'journalctl -f -n 0 -u ceph-75c37498-de77-11ee-95c7-87774f69a715@osd.3.service' 2024-03-10T00:54:51.242 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-10T00:54:51.243 INFO:tasks.cephadm.osd.3:Stopped osd.3 2024-03-10T00:54:51.243 INFO:tasks.cephadm.osd.4:Stopping osd.4... 2024-03-10T00:54:51.243 DEBUG:teuthology.orchestra.run.smithi170:> sudo systemctl stop ceph-75c37498-de77-11ee-95c7-87774f69a715@osd.4 2024-03-10T00:54:51.478 INFO:journalctl@ceph.osd.4.smithi170.stdout:Mar 10 00:54:51 smithi170 systemd[1]: Stopping Ceph osd.4 for 75c37498-de77-11ee-95c7-87774f69a715... 2024-03-10T00:54:51.478 INFO:journalctl@ceph.osd.6.smithi170.stdout:Mar 10 00:54:51 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-osd-6[43049]: 2024-03-10T00:54:51.349+0000 7f5852423700 -1 osd.6 71 heartbeat_check: no reply from 172.21.15.106:6806 osd.0 since back 2024-03-10T00:54:17.852283+0000 front 2024-03-10T00:54:17.852086+0000 (oldest deadline 2024-03-10T00:54:43.752263+0000) 2024-03-10T00:54:51.479 INFO:journalctl@ceph.osd.7.smithi170.stdout:Mar 10 00:54:51 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-osd-7[47108]: 2024-03-10T00:54:51.211+0000 7f7b81779700 -1 osd.7 71 heartbeat_check: no reply from 172.21.15.106:6806 osd.0 since back 2024-03-10T00:54:19.432267+0000 front 2024-03-10T00:54:19.432106+0000 (oldest deadline 2024-03-10T00:54:43.532104+0000) 2024-03-10T00:54:51.774 INFO:journalctl@ceph.osd.4.smithi170.stdout:Mar 10 00:54:51 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-osd-4[35111]: 2024-03-10T00:54:51.552+0000 7f053cb3d700 -1 osd.4 71 heartbeat_check: no reply from 172.21.15.106:6806 osd.0 since back 2024-03-10T00:54:19.067553+0000 front 2024-03-10T00:54:19.067498+0000 (oldest deadline 2024-03-10T00:54:44.967307+0000) 2024-03-10T00:54:51.774 INFO:journalctl@ceph.osd.5.smithi170.stdout:Mar 10 00:54:51 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-osd-5[38986]: 2024-03-10T00:54:51.480+0000 7f72444e9700 -1 osd.5 71 heartbeat_check: no reply from 172.21.15.106:6806 osd.0 since back 2024-03-10T00:54:18.209644+0000 front 2024-03-10T00:54:18.209502+0000 (oldest deadline 2024-03-10T00:54:48.210011+0000) 2024-03-10T00:54:52.475 INFO:journalctl@ceph.osd.6.smithi170.stdout:Mar 10 00:54:52 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-osd-6[43049]: 2024-03-10T00:54:52.387+0000 7f5852423700 -1 osd.6 71 heartbeat_check: no reply from 172.21.15.106:6806 osd.0 since back 2024-03-10T00:54:17.852283+0000 front 2024-03-10T00:54:17.852086+0000 (oldest deadline 2024-03-10T00:54:43.752263+0000) 2024-03-10T00:54:52.476 INFO:journalctl@ceph.osd.7.smithi170.stdout:Mar 10 00:54:52 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-osd-7[47108]: 2024-03-10T00:54:52.198+0000 7f7b81779700 -1 osd.7 71 heartbeat_check: no reply from 172.21.15.106:6806 osd.0 since back 2024-03-10T00:54:19.432267+0000 front 2024-03-10T00:54:19.432106+0000 (oldest deadline 2024-03-10T00:54:43.532104+0000) 2024-03-10T00:54:52.476 INFO:journalctl@ceph.osd.4.smithi170.stdout:Mar 10 00:54:52 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-osd-4[35111]: 2024-03-10T00:54:52.303+0000 7f0540549700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.4 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-10T00:54:52.476 INFO:journalctl@ceph.osd.4.smithi170.stdout:Mar 10 00:54:52 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-osd-4[35111]: 2024-03-10T00:54:52.303+0000 7f0540549700 -1 osd.4 71 *** Got signal Terminated *** 2024-03-10T00:54:52.476 INFO:journalctl@ceph.osd.4.smithi170.stdout:Mar 10 00:54:52 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-osd-4[35111]: 2024-03-10T00:54:52.303+0000 7f0540549700 -1 osd.4 71 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-10T00:54:52.774 INFO:journalctl@ceph.osd.5.smithi170.stdout:Mar 10 00:54:52 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-osd-5[38986]: 2024-03-10T00:54:52.477+0000 7f72444e9700 -1 osd.5 71 heartbeat_check: no reply from 172.21.15.106:6806 osd.0 since back 2024-03-10T00:54:18.209644+0000 front 2024-03-10T00:54:18.209502+0000 (oldest deadline 2024-03-10T00:54:48.210011+0000) 2024-03-10T00:54:52.774 INFO:journalctl@ceph.osd.4.smithi170.stdout:Mar 10 00:54:52 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-osd-4[35111]: 2024-03-10T00:54:52.560+0000 7f053cb3d700 -1 osd.4 71 heartbeat_check: no reply from 172.21.15.106:6806 osd.0 since back 2024-03-10T00:54:19.067553+0000 front 2024-03-10T00:54:19.067498+0000 (oldest deadline 2024-03-10T00:54:44.967307+0000) 2024-03-10T00:54:53.441 INFO:journalctl@ceph.osd.6.smithi170.stdout:Mar 10 00:54:53 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-osd-6[43049]: 2024-03-10T00:54:53.382+0000 7f5852423700 -1 osd.6 71 heartbeat_check: no reply from 172.21.15.106:6806 osd.0 since back 2024-03-10T00:54:17.852283+0000 front 2024-03-10T00:54:17.852086+0000 (oldest deadline 2024-03-10T00:54:43.752263+0000) 2024-03-10T00:54:53.441 INFO:journalctl@ceph.osd.7.smithi170.stdout:Mar 10 00:54:53 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-osd-7[47108]: 2024-03-10T00:54:53.192+0000 7f7b81779700 -1 osd.7 71 heartbeat_check: no reply from 172.21.15.106:6806 osd.0 since back 2024-03-10T00:54:19.432267+0000 front 2024-03-10T00:54:19.432106+0000 (oldest deadline 2024-03-10T00:54:43.532104+0000) 2024-03-10T00:54:53.774 INFO:journalctl@ceph.osd.5.smithi170.stdout:Mar 10 00:54:53 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-osd-5[38986]: 2024-03-10T00:54:53.493+0000 7f72444e9700 -1 osd.5 71 heartbeat_check: no reply from 172.21.15.106:6806 osd.0 since back 2024-03-10T00:54:18.209644+0000 front 2024-03-10T00:54:18.209502+0000 (oldest deadline 2024-03-10T00:54:48.210011+0000) 2024-03-10T00:54:53.774 INFO:journalctl@ceph.osd.5.smithi170.stdout:Mar 10 00:54:53 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-osd-5[38986]: 2024-03-10T00:54:53.493+0000 7f72444e9700 -1 osd.5 71 heartbeat_check: no reply from 172.21.15.106:6814 osd.1 since back 2024-03-10T00:54:28.210388+0000 front 2024-03-10T00:54:28.210493+0000 (oldest deadline 2024-03-10T00:54:52.910325+0000) 2024-03-10T00:54:53.774 INFO:journalctl@ceph.osd.4.smithi170.stdout:Mar 10 00:54:53 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-osd-4[35111]: 2024-03-10T00:54:53.566+0000 7f053cb3d700 -1 osd.4 71 heartbeat_check: no reply from 172.21.15.106:6806 osd.0 since back 2024-03-10T00:54:19.067553+0000 front 2024-03-10T00:54:19.067498+0000 (oldest deadline 2024-03-10T00:54:44.967307+0000) 2024-03-10T00:54:53.774 INFO:journalctl@ceph.osd.4.smithi170.stdout:Mar 10 00:54:53 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-osd-4[35111]: 2024-03-10T00:54:53.566+0000 7f053cb3d700 -1 osd.4 71 heartbeat_check: no reply from 172.21.15.106:6814 osd.1 since back 2024-03-10T00:54:27.669457+0000 front 2024-03-10T00:54:27.669443+0000 (oldest deadline 2024-03-10T00:54:52.969383+0000) 2024-03-10T00:54:54.480 INFO:journalctl@ceph.osd.7.smithi170.stdout:Mar 10 00:54:54 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-osd-7[47108]: 2024-03-10T00:54:54.173+0000 7f7b81779700 -1 osd.7 71 heartbeat_check: no reply from 172.21.15.106:6806 osd.0 since back 2024-03-10T00:54:19.432267+0000 front 2024-03-10T00:54:19.432106+0000 (oldest deadline 2024-03-10T00:54:43.532104+0000) 2024-03-10T00:54:54.480 INFO:journalctl@ceph.osd.6.smithi170.stdout:Mar 10 00:54:54 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-osd-6[43049]: 2024-03-10T00:54:54.341+0000 7f5852423700 -1 osd.6 71 heartbeat_check: no reply from 172.21.15.106:6806 osd.0 since back 2024-03-10T00:54:17.852283+0000 front 2024-03-10T00:54:17.852086+0000 (oldest deadline 2024-03-10T00:54:43.752263+0000) 2024-03-10T00:54:54.480 INFO:journalctl@ceph.osd.6.smithi170.stdout:Mar 10 00:54:54 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-osd-6[43049]: 2024-03-10T00:54:54.341+0000 7f5852423700 -1 osd.6 71 heartbeat_check: no reply from 172.21.15.106:6814 osd.1 since back 2024-03-10T00:54:28.453247+0000 front 2024-03-10T00:54:28.453219+0000 (oldest deadline 2024-03-10T00:54:53.753144+0000) 2024-03-10T00:54:54.774 INFO:journalctl@ceph.osd.5.smithi170.stdout:Mar 10 00:54:54 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-osd-5[38986]: 2024-03-10T00:54:54.482+0000 7f72444e9700 -1 osd.5 71 heartbeat_check: no reply from 172.21.15.106:6806 osd.0 since back 2024-03-10T00:54:18.209644+0000 front 2024-03-10T00:54:18.209502+0000 (oldest deadline 2024-03-10T00:54:48.210011+0000) 2024-03-10T00:54:54.774 INFO:journalctl@ceph.osd.5.smithi170.stdout:Mar 10 00:54:54 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-osd-5[38986]: 2024-03-10T00:54:54.482+0000 7f72444e9700 -1 osd.5 71 heartbeat_check: no reply from 172.21.15.106:6814 osd.1 since back 2024-03-10T00:54:28.210388+0000 front 2024-03-10T00:54:28.210493+0000 (oldest deadline 2024-03-10T00:54:52.910325+0000) 2024-03-10T00:54:54.774 INFO:journalctl@ceph.osd.4.smithi170.stdout:Mar 10 00:54:54 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-osd-4[35111]: 2024-03-10T00:54:54.613+0000 7f053cb3d700 -1 osd.4 71 heartbeat_check: no reply from 172.21.15.106:6806 osd.0 since back 2024-03-10T00:54:19.067553+0000 front 2024-03-10T00:54:19.067498+0000 (oldest deadline 2024-03-10T00:54:44.967307+0000) 2024-03-10T00:54:54.774 INFO:journalctl@ceph.osd.4.smithi170.stdout:Mar 10 00:54:54 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-osd-4[35111]: 2024-03-10T00:54:54.613+0000 7f053cb3d700 -1 osd.4 71 heartbeat_check: no reply from 172.21.15.106:6814 osd.1 since back 2024-03-10T00:54:27.669457+0000 front 2024-03-10T00:54:27.669443+0000 (oldest deadline 2024-03-10T00:54:52.969383+0000) 2024-03-10T00:54:55.432 INFO:journalctl@ceph.osd.6.smithi170.stdout:Mar 10 00:54:55 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-osd-6[43049]: 2024-03-10T00:54:55.351+0000 7f5852423700 -1 osd.6 71 heartbeat_check: no reply from 172.21.15.106:6806 osd.0 since back 2024-03-10T00:54:17.852283+0000 front 2024-03-10T00:54:17.852086+0000 (oldest deadline 2024-03-10T00:54:43.752263+0000) 2024-03-10T00:54:55.433 INFO:journalctl@ceph.osd.6.smithi170.stdout:Mar 10 00:54:55 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-osd-6[43049]: 2024-03-10T00:54:55.351+0000 7f5852423700 -1 osd.6 71 heartbeat_check: no reply from 172.21.15.106:6814 osd.1 since back 2024-03-10T00:54:28.453247+0000 front 2024-03-10T00:54:28.453219+0000 (oldest deadline 2024-03-10T00:54:53.753144+0000) 2024-03-10T00:54:55.433 INFO:journalctl@ceph.osd.7.smithi170.stdout:Mar 10 00:54:55 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-osd-7[47108]: 2024-03-10T00:54:55.184+0000 7f7b81779700 -1 osd.7 71 heartbeat_check: no reply from 172.21.15.106:6806 osd.0 since back 2024-03-10T00:54:19.432267+0000 front 2024-03-10T00:54:19.432106+0000 (oldest deadline 2024-03-10T00:54:43.532104+0000) 2024-03-10T00:54:55.433 INFO:journalctl@ceph.osd.7.smithi170.stdout:Mar 10 00:54:55 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-osd-7[47108]: 2024-03-10T00:54:55.184+0000 7f7b81779700 -1 osd.7 71 heartbeat_check: no reply from 172.21.15.106:6814 osd.1 since back 2024-03-10T00:54:30.533390+0000 front 2024-03-10T00:54:30.533458+0000 (oldest deadline 2024-03-10T00:54:54.633272+0000) 2024-03-10T00:54:55.774 INFO:journalctl@ceph.osd.4.smithi170.stdout:Mar 10 00:54:55 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-osd-4[35111]: 2024-03-10T00:54:55.640+0000 7f053cb3d700 -1 osd.4 71 heartbeat_check: no reply from 172.21.15.106:6806 osd.0 since back 2024-03-10T00:54:19.067553+0000 front 2024-03-10T00:54:19.067498+0000 (oldest deadline 2024-03-10T00:54:44.967307+0000) 2024-03-10T00:54:55.774 INFO:journalctl@ceph.osd.4.smithi170.stdout:Mar 10 00:54:55 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-osd-4[35111]: 2024-03-10T00:54:55.640+0000 7f053cb3d700 -1 osd.4 71 heartbeat_check: no reply from 172.21.15.106:6814 osd.1 since back 2024-03-10T00:54:27.669457+0000 front 2024-03-10T00:54:27.669443+0000 (oldest deadline 2024-03-10T00:54:52.969383+0000) 2024-03-10T00:54:55.774 INFO:journalctl@ceph.osd.5.smithi170.stdout:Mar 10 00:54:55 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-osd-5[38986]: 2024-03-10T00:54:55.499+0000 7f72444e9700 -1 osd.5 71 heartbeat_check: no reply from 172.21.15.106:6806 osd.0 since back 2024-03-10T00:54:18.209644+0000 front 2024-03-10T00:54:18.209502+0000 (oldest deadline 2024-03-10T00:54:48.210011+0000) 2024-03-10T00:54:55.774 INFO:journalctl@ceph.osd.5.smithi170.stdout:Mar 10 00:54:55 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-osd-5[38986]: 2024-03-10T00:54:55.499+0000 7f72444e9700 -1 osd.5 71 heartbeat_check: no reply from 172.21.15.106:6814 osd.1 since back 2024-03-10T00:54:28.210388+0000 front 2024-03-10T00:54:28.210493+0000 (oldest deadline 2024-03-10T00:54:52.910325+0000) 2024-03-10T00:54:56.326 INFO:journalctl@ceph.osd.7.smithi170.stdout:Mar 10 00:54:56 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-osd-7[47108]: 2024-03-10T00:54:56.157+0000 7f7b81779700 -1 osd.7 71 heartbeat_check: no reply from 172.21.15.106:6806 osd.0 since back 2024-03-10T00:54:19.432267+0000 front 2024-03-10T00:54:19.432106+0000 (oldest deadline 2024-03-10T00:54:43.532104+0000) 2024-03-10T00:54:56.326 INFO:journalctl@ceph.osd.7.smithi170.stdout:Mar 10 00:54:56 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-osd-7[47108]: 2024-03-10T00:54:56.157+0000 7f7b81779700 -1 osd.7 71 heartbeat_check: no reply from 172.21.15.106:6814 osd.1 since back 2024-03-10T00:54:30.533390+0000 front 2024-03-10T00:54:30.533458+0000 (oldest deadline 2024-03-10T00:54:54.633272+0000) 2024-03-10T00:54:56.660 INFO:journalctl@ceph.osd.5.smithi170.stdout:Mar 10 00:54:56 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-osd-5[38986]: 2024-03-10T00:54:56.472+0000 7f72444e9700 -1 osd.5 71 heartbeat_check: no reply from 172.21.15.106:6806 osd.0 since back 2024-03-10T00:54:18.209644+0000 front 2024-03-10T00:54:18.209502+0000 (oldest deadline 2024-03-10T00:54:48.210011+0000) 2024-03-10T00:54:56.660 INFO:journalctl@ceph.osd.5.smithi170.stdout:Mar 10 00:54:56 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-osd-5[38986]: 2024-03-10T00:54:56.472+0000 7f72444e9700 -1 osd.5 71 heartbeat_check: no reply from 172.21.15.106:6814 osd.1 since back 2024-03-10T00:54:28.210388+0000 front 2024-03-10T00:54:28.210493+0000 (oldest deadline 2024-03-10T00:54:52.910325+0000) 2024-03-10T00:54:56.660 INFO:journalctl@ceph.osd.6.smithi170.stdout:Mar 10 00:54:56 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-osd-6[43049]: 2024-03-10T00:54:56.328+0000 7f5852423700 -1 osd.6 71 heartbeat_check: no reply from 172.21.15.106:6806 osd.0 since back 2024-03-10T00:54:17.852283+0000 front 2024-03-10T00:54:17.852086+0000 (oldest deadline 2024-03-10T00:54:43.752263+0000) 2024-03-10T00:54:56.660 INFO:journalctl@ceph.osd.6.smithi170.stdout:Mar 10 00:54:56 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-osd-6[43049]: 2024-03-10T00:54:56.328+0000 7f5852423700 -1 osd.6 71 heartbeat_check: no reply from 172.21.15.106:6814 osd.1 since back 2024-03-10T00:54:28.453247+0000 front 2024-03-10T00:54:28.453219+0000 (oldest deadline 2024-03-10T00:54:53.753144+0000) 2024-03-10T00:54:57.024 INFO:journalctl@ceph.osd.4.smithi170.stdout:Mar 10 00:54:56 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-osd-4[35111]: 2024-03-10T00:54:56.662+0000 7f053cb3d700 -1 osd.4 71 heartbeat_check: no reply from 172.21.15.106:6806 osd.0 since back 2024-03-10T00:54:19.067553+0000 front 2024-03-10T00:54:19.067498+0000 (oldest deadline 2024-03-10T00:54:44.967307+0000) 2024-03-10T00:54:57.024 INFO:journalctl@ceph.osd.4.smithi170.stdout:Mar 10 00:54:56 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-osd-4[35111]: 2024-03-10T00:54:56.662+0000 7f053cb3d700 -1 osd.4 71 heartbeat_check: no reply from 172.21.15.106:6814 osd.1 since back 2024-03-10T00:54:27.669457+0000 front 2024-03-10T00:54:27.669443+0000 (oldest deadline 2024-03-10T00:54:52.969383+0000) 2024-03-10T00:54:57.440 INFO:journalctl@ceph.osd.6.smithi170.stdout:Mar 10 00:54:57 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-osd-6[43049]: 2024-03-10T00:54:57.328+0000 7f5852423700 -1 osd.6 71 heartbeat_check: no reply from 172.21.15.106:6806 osd.0 since back 2024-03-10T00:54:17.852283+0000 front 2024-03-10T00:54:17.852086+0000 (oldest deadline 2024-03-10T00:54:43.752263+0000) 2024-03-10T00:54:57.440 INFO:journalctl@ceph.osd.6.smithi170.stdout:Mar 10 00:54:57 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-osd-6[43049]: 2024-03-10T00:54:57.328+0000 7f5852423700 -1 osd.6 71 heartbeat_check: no reply from 172.21.15.106:6814 osd.1 since back 2024-03-10T00:54:28.453247+0000 front 2024-03-10T00:54:28.453219+0000 (oldest deadline 2024-03-10T00:54:53.753144+0000) 2024-03-10T00:54:57.440 INFO:journalctl@ceph.osd.7.smithi170.stdout:Mar 10 00:54:57 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-osd-7[47108]: 2024-03-10T00:54:57.123+0000 7f7b81779700 -1 osd.7 71 heartbeat_check: no reply from 172.21.15.106:6806 osd.0 since back 2024-03-10T00:54:19.432267+0000 front 2024-03-10T00:54:19.432106+0000 (oldest deadline 2024-03-10T00:54:43.532104+0000) 2024-03-10T00:54:57.440 INFO:journalctl@ceph.osd.7.smithi170.stdout:Mar 10 00:54:57 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-osd-7[47108]: 2024-03-10T00:54:57.123+0000 7f7b81779700 -1 osd.7 71 heartbeat_check: no reply from 172.21.15.106:6814 osd.1 since back 2024-03-10T00:54:30.533390+0000 front 2024-03-10T00:54:30.533458+0000 (oldest deadline 2024-03-10T00:54:54.633272+0000) 2024-03-10T00:54:57.714 INFO:journalctl@ceph.osd.5.smithi170.stdout:Mar 10 00:54:57 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-osd-5[38986]: 2024-03-10T00:54:57.443+0000 7f72444e9700 -1 osd.5 71 heartbeat_check: no reply from 172.21.15.106:6806 osd.0 since back 2024-03-10T00:54:18.209644+0000 front 2024-03-10T00:54:18.209502+0000 (oldest deadline 2024-03-10T00:54:48.210011+0000) 2024-03-10T00:54:57.715 INFO:journalctl@ceph.osd.5.smithi170.stdout:Mar 10 00:54:57 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-osd-5[38986]: 2024-03-10T00:54:57.443+0000 7f72444e9700 -1 osd.5 71 heartbeat_check: no reply from 172.21.15.106:6814 osd.1 since back 2024-03-10T00:54:28.210388+0000 front 2024-03-10T00:54:28.210493+0000 (oldest deadline 2024-03-10T00:54:52.910325+0000) 2024-03-10T00:54:58.019 INFO:journalctl@ceph.osd.4.smithi170.stdout:Mar 10 00:54:57 smithi170 podman[66034]: 2024-03-10 00:54:57.717923641 +0000 UTC m=+5.518938664 container died 895388359a4ea67cb874a7e9a433fe8c9b0bb84e779404853c492fa1d00756f0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bc4a24690ce5d7255d490e21058d8c114b55e1b93a3d8345f90b6700ad29dada, name=ceph-75c37498-de77-11ee-95c7-87774f69a715-osd-4, GIT_CLEAN=True, ceph=True, RELEASE=wip-yuri4-testing-2024-03-05-0854-831bc18, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.build-date=20240222) 2024-03-10T00:54:58.274 INFO:journalctl@ceph.osd.4.smithi170.stdout:Mar 10 00:54:58 smithi170 podman[66034]: 2024-03-10 00:54:58.213553661 +0000 UTC m=+6.014568740 container cleanup 895388359a4ea67cb874a7e9a433fe8c9b0bb84e779404853c492fa1d00756f0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bc4a24690ce5d7255d490e21058d8c114b55e1b93a3d8345f90b6700ad29dada, name=ceph-75c37498-de77-11ee-95c7-87774f69a715-osd-4, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, ceph=True, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, RELEASE=wip-yuri4-testing-2024-03-05-0854-831bc18, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240222, GIT_CLEAN=True) 2024-03-10T00:54:58.274 INFO:journalctl@ceph.osd.4.smithi170.stdout:Mar 10 00:54:58 smithi170 bash[66034]: ceph-75c37498-de77-11ee-95c7-87774f69a715-osd-4 2024-03-10T00:54:58.274 INFO:journalctl@ceph.osd.7.smithi170.stdout:Mar 10 00:54:58 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-osd-7[47108]: 2024-03-10T00:54:58.104+0000 7f7b81779700 -1 osd.7 71 heartbeat_check: no reply from 172.21.15.106:6806 osd.0 since back 2024-03-10T00:54:19.432267+0000 front 2024-03-10T00:54:19.432106+0000 (oldest deadline 2024-03-10T00:54:43.532104+0000) 2024-03-10T00:54:58.274 INFO:journalctl@ceph.osd.7.smithi170.stdout:Mar 10 00:54:58 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-osd-7[47108]: 2024-03-10T00:54:58.104+0000 7f7b81779700 -1 osd.7 71 heartbeat_check: no reply from 172.21.15.106:6814 osd.1 since back 2024-03-10T00:54:30.533390+0000 front 2024-03-10T00:54:30.533458+0000 (oldest deadline 2024-03-10T00:54:54.633272+0000) 2024-03-10T00:54:58.750 INFO:journalctl@ceph.osd.4.smithi170.stdout:Mar 10 00:54:58 smithi170 podman[66144]: 2024-03-10 00:54:58.440031302 +0000 UTC m=+0.719848323 container remove 895388359a4ea67cb874a7e9a433fe8c9b0bb84e779404853c492fa1d00756f0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bc4a24690ce5d7255d490e21058d8c114b55e1b93a3d8345f90b6700ad29dada, name=ceph-75c37498-de77-11ee-95c7-87774f69a715-osd-4, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, io.buildah.version=1.33.5, RELEASE=wip-yuri4-testing-2024-03-05-0854-831bc18, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240222, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS) 2024-03-10T00:54:58.750 INFO:journalctl@ceph.osd.5.smithi170.stdout:Mar 10 00:54:58 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-osd-5[38986]: 2024-03-10T00:54:58.490+0000 7f72444e9700 -1 osd.5 71 heartbeat_check: no reply from 172.21.15.106:6806 osd.0 since back 2024-03-10T00:54:18.209644+0000 front 2024-03-10T00:54:18.209502+0000 (oldest deadline 2024-03-10T00:54:48.210011+0000) 2024-03-10T00:54:58.751 INFO:journalctl@ceph.osd.5.smithi170.stdout:Mar 10 00:54:58 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-osd-5[38986]: 2024-03-10T00:54:58.490+0000 7f72444e9700 -1 osd.5 71 heartbeat_check: no reply from 172.21.15.106:6814 osd.1 since back 2024-03-10T00:54:28.210388+0000 front 2024-03-10T00:54:28.210493+0000 (oldest deadline 2024-03-10T00:54:52.910325+0000) 2024-03-10T00:54:58.751 INFO:journalctl@ceph.osd.6.smithi170.stdout:Mar 10 00:54:58 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-osd-6[43049]: 2024-03-10T00:54:58.366+0000 7f5852423700 -1 osd.6 71 heartbeat_check: no reply from 172.21.15.106:6806 osd.0 since back 2024-03-10T00:54:17.852283+0000 front 2024-03-10T00:54:17.852086+0000 (oldest deadline 2024-03-10T00:54:43.752263+0000) 2024-03-10T00:54:58.751 INFO:journalctl@ceph.osd.6.smithi170.stdout:Mar 10 00:54:58 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-osd-6[43049]: 2024-03-10T00:54:58.366+0000 7f5852423700 -1 osd.6 71 heartbeat_check: no reply from 172.21.15.106:6814 osd.1 since back 2024-03-10T00:54:28.453247+0000 front 2024-03-10T00:54:28.453219+0000 (oldest deadline 2024-03-10T00:54:53.753144+0000) 2024-03-10T00:54:59.005 INFO:journalctl@ceph.osd.4.smithi170.stdout:Mar 10 00:54:58 smithi170 podman[66242]: 2024-03-10 00:54:58.656005213 +0000 UTC m=+0.018359947 image pull 01a4f6a6bb3e92ab9ab39a2f835684ea9b10e6ec9fb81ac08ebea4165583a538 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bc4a24690ce5d7255d490e21058d8c114b55e1b93a3d8345f90b6700ad29dada 2024-03-10T00:54:59.005 INFO:journalctl@ceph.osd.4.smithi170.stdout:Mar 10 00:54:58 smithi170 podman[66242]: 2024-03-10 00:54:58.794588594 +0000 UTC m=+0.156943327 container create d07bb3b614c780ba7c643b0dad80e0a6a40b1e92bc5baa60363ce23c69bb2440 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bc4a24690ce5d7255d490e21058d8c114b55e1b93a3d8345f90b6700ad29dada, name=ceph-75c37498-de77-11ee-95c7-87774f69a715-osd-4-deactivate, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, ceph=True, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, org.label-schema.build-date=20240222, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, RELEASE=wip-yuri4-testing-2024-03-05-0854-831bc18, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5) 2024-03-10T00:54:59.005 INFO:journalctl@ceph.osd.4.smithi170.stdout:Mar 10 00:54:59 smithi170 podman[66242]: 2024-03-10 00:54:59.001984979 +0000 UTC m=+0.364339733 container init d07bb3b614c780ba7c643b0dad80e0a6a40b1e92bc5baa60363ce23c69bb2440 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bc4a24690ce5d7255d490e21058d8c114b55e1b93a3d8345f90b6700ad29dada, name=ceph-75c37498-de77-11ee-95c7-87774f69a715-osd-4-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.build-date=20240222, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, RELEASE=wip-yuri4-testing-2024-03-05-0854-831bc18) 2024-03-10T00:54:59.274 INFO:journalctl@ceph.osd.4.smithi170.stdout:Mar 10 00:54:59 smithi170 podman[66242]: 2024-03-10 00:54:59.007866993 +0000 UTC m=+0.370221727 container start d07bb3b614c780ba7c643b0dad80e0a6a40b1e92bc5baa60363ce23c69bb2440 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bc4a24690ce5d7255d490e21058d8c114b55e1b93a3d8345f90b6700ad29dada, name=ceph-75c37498-de77-11ee-95c7-87774f69a715-osd-4-deactivate, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, org.label-schema.build-date=20240222, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, RELEASE=wip-yuri4-testing-2024-03-05-0854-831bc18, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS) 2024-03-10T00:54:59.274 INFO:journalctl@ceph.osd.4.smithi170.stdout:Mar 10 00:54:59 smithi170 podman[66242]: 2024-03-10 00:54:59.130574332 +0000 UTC m=+0.492929068 container attach d07bb3b614c780ba7c643b0dad80e0a6a40b1e92bc5baa60363ce23c69bb2440 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bc4a24690ce5d7255d490e21058d8c114b55e1b93a3d8345f90b6700ad29dada, name=ceph-75c37498-de77-11ee-95c7-87774f69a715-osd-4-deactivate, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, RELEASE=wip-yuri4-testing-2024-03-05-0854-831bc18, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240222, GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux ) 2024-03-10T00:54:59.274 INFO:journalctl@ceph.osd.7.smithi170.stdout:Mar 10 00:54:59 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-osd-7[47108]: 2024-03-10T00:54:59.129+0000 7f7b81779700 -1 osd.7 71 heartbeat_check: no reply from 172.21.15.106:6806 osd.0 since back 2024-03-10T00:54:19.432267+0000 front 2024-03-10T00:54:19.432106+0000 (oldest deadline 2024-03-10T00:54:43.532104+0000) 2024-03-10T00:54:59.274 INFO:journalctl@ceph.osd.7.smithi170.stdout:Mar 10 00:54:59 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-osd-7[47108]: 2024-03-10T00:54:59.129+0000 7f7b81779700 -1 osd.7 71 heartbeat_check: no reply from 172.21.15.106:6814 osd.1 since back 2024-03-10T00:54:30.533390+0000 front 2024-03-10T00:54:30.533458+0000 (oldest deadline 2024-03-10T00:54:54.633272+0000) 2024-03-10T00:54:59.610 INFO:journalctl@ceph.osd.4.smithi170.stdout:Mar 10 00:54:59 smithi170 podman[66242]: 2024-03-10 00:54:59.301075827 +0000 UTC m=+0.663430553 container died d07bb3b614c780ba7c643b0dad80e0a6a40b1e92bc5baa60363ce23c69bb2440 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bc4a24690ce5d7255d490e21058d8c114b55e1b93a3d8345f90b6700ad29dada, name=ceph-75c37498-de77-11ee-95c7-87774f69a715-osd-4-deactivate, maintainer=Guillaume Abrioux , RELEASE=wip-yuri4-testing-2024-03-05-0854-831bc18, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, org.label-schema.license=GPLv2, ceph=True, org.label-schema.build-date=20240222, io.buildah.version=1.33.5) 2024-03-10T00:54:59.610 INFO:journalctl@ceph.osd.5.smithi170.stdout:Mar 10 00:54:59 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-osd-5[38986]: 2024-03-10T00:54:59.497+0000 7f72444e9700 -1 osd.5 71 heartbeat_check: no reply from 172.21.15.106:6806 osd.0 since back 2024-03-10T00:54:18.209644+0000 front 2024-03-10T00:54:18.209502+0000 (oldest deadline 2024-03-10T00:54:48.210011+0000) 2024-03-10T00:54:59.610 INFO:journalctl@ceph.osd.5.smithi170.stdout:Mar 10 00:54:59 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-osd-5[38986]: 2024-03-10T00:54:59.497+0000 7f72444e9700 -1 osd.5 71 heartbeat_check: no reply from 172.21.15.106:6814 osd.1 since back 2024-03-10T00:54:28.210388+0000 front 2024-03-10T00:54:28.210493+0000 (oldest deadline 2024-03-10T00:54:52.910325+0000) 2024-03-10T00:54:59.611 INFO:journalctl@ceph.osd.6.smithi170.stdout:Mar 10 00:54:59 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-osd-6[43049]: 2024-03-10T00:54:59.345+0000 7f5852423700 -1 osd.6 71 heartbeat_check: no reply from 172.21.15.106:6806 osd.0 since back 2024-03-10T00:54:17.852283+0000 front 2024-03-10T00:54:17.852086+0000 (oldest deadline 2024-03-10T00:54:43.752263+0000) 2024-03-10T00:54:59.611 INFO:journalctl@ceph.osd.6.smithi170.stdout:Mar 10 00:54:59 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-osd-6[43049]: 2024-03-10T00:54:59.345+0000 7f5852423700 -1 osd.6 71 heartbeat_check: no reply from 172.21.15.106:6814 osd.1 since back 2024-03-10T00:54:28.453247+0000 front 2024-03-10T00:54:28.453219+0000 (oldest deadline 2024-03-10T00:54:53.753144+0000) 2024-03-10T00:54:59.930 DEBUG:teuthology.orchestra.run.smithi170:> sudo pkill -f 'journalctl -f -n 0 -u ceph-75c37498-de77-11ee-95c7-87774f69a715@osd.4.service' 2024-03-10T00:54:59.974 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-10T00:54:59.974 INFO:tasks.cephadm.osd.4:Stopped osd.4 2024-03-10T00:54:59.975 INFO:tasks.cephadm.osd.5:Stopping osd.5... 2024-03-10T00:54:59.975 DEBUG:teuthology.orchestra.run.smithi170:> sudo systemctl stop ceph-75c37498-de77-11ee-95c7-87774f69a715@osd.5 2024-03-10T00:55:00.274 INFO:journalctl@ceph.osd.5.smithi170.stdout:Mar 10 00:55:00 smithi170 systemd[1]: Stopping Ceph osd.5 for 75c37498-de77-11ee-95c7-87774f69a715... 2024-03-10T00:55:00.274 INFO:journalctl@ceph.osd.7.smithi170.stdout:Mar 10 00:55:00 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-osd-7[47108]: 2024-03-10T00:55:00.092+0000 7f7b81779700 -1 osd.7 71 heartbeat_check: no reply from 172.21.15.106:6806 osd.0 since back 2024-03-10T00:54:19.432267+0000 front 2024-03-10T00:54:19.432106+0000 (oldest deadline 2024-03-10T00:54:43.532104+0000) 2024-03-10T00:55:00.274 INFO:journalctl@ceph.osd.7.smithi170.stdout:Mar 10 00:55:00 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-osd-7[47108]: 2024-03-10T00:55:00.092+0000 7f7b81779700 -1 osd.7 71 heartbeat_check: no reply from 172.21.15.106:6814 osd.1 since back 2024-03-10T00:54:30.533390+0000 front 2024-03-10T00:54:30.533458+0000 (oldest deadline 2024-03-10T00:54:54.633272+0000) 2024-03-10T00:55:00.774 INFO:journalctl@ceph.osd.6.smithi170.stdout:Mar 10 00:55:00 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-osd-6[43049]: 2024-03-10T00:55:00.331+0000 7f5852423700 -1 osd.6 71 heartbeat_check: no reply from 172.21.15.106:6806 osd.0 since back 2024-03-10T00:54:17.852283+0000 front 2024-03-10T00:54:17.852086+0000 (oldest deadline 2024-03-10T00:54:43.752263+0000) 2024-03-10T00:55:00.774 INFO:journalctl@ceph.osd.6.smithi170.stdout:Mar 10 00:55:00 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-osd-6[43049]: 2024-03-10T00:55:00.331+0000 7f5852423700 -1 osd.6 71 heartbeat_check: no reply from 172.21.15.106:6814 osd.1 since back 2024-03-10T00:54:28.453247+0000 front 2024-03-10T00:54:28.453219+0000 (oldest deadline 2024-03-10T00:54:53.753144+0000) 2024-03-10T00:55:00.774 INFO:journalctl@ceph.osd.5.smithi170.stdout:Mar 10 00:55:00 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-osd-5[38986]: 2024-03-10T00:55:00.307+0000 7f724befd700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.5 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-10T00:55:00.774 INFO:journalctl@ceph.osd.5.smithi170.stdout:Mar 10 00:55:00 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-osd-5[38986]: 2024-03-10T00:55:00.307+0000 7f724befd700 -1 osd.5 71 *** Got signal Terminated *** 2024-03-10T00:55:00.774 INFO:journalctl@ceph.osd.5.smithi170.stdout:Mar 10 00:55:00 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-osd-5[38986]: 2024-03-10T00:55:00.307+0000 7f724befd700 -1 osd.5 71 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-10T00:55:00.775 INFO:journalctl@ceph.osd.5.smithi170.stdout:Mar 10 00:55:00 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-osd-5[38986]: 2024-03-10T00:55:00.547+0000 7f72444e9700 -1 osd.5 71 heartbeat_check: no reply from 172.21.15.106:6806 osd.0 since back 2024-03-10T00:54:18.209644+0000 front 2024-03-10T00:54:18.209502+0000 (oldest deadline 2024-03-10T00:54:48.210011+0000) 2024-03-10T00:55:00.775 INFO:journalctl@ceph.osd.5.smithi170.stdout:Mar 10 00:55:00 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-osd-5[38986]: 2024-03-10T00:55:00.547+0000 7f72444e9700 -1 osd.5 71 heartbeat_check: no reply from 172.21.15.106:6814 osd.1 since back 2024-03-10T00:54:28.210388+0000 front 2024-03-10T00:54:28.210493+0000 (oldest deadline 2024-03-10T00:54:52.910325+0000) 2024-03-10T00:55:01.524 INFO:journalctl@ceph.osd.6.smithi170.stdout:Mar 10 00:55:01 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-osd-6[43049]: 2024-03-10T00:55:01.300+0000 7f5852423700 -1 osd.6 71 heartbeat_check: no reply from 172.21.15.106:6806 osd.0 since back 2024-03-10T00:54:17.852283+0000 front 2024-03-10T00:54:17.852086+0000 (oldest deadline 2024-03-10T00:54:43.752263+0000) 2024-03-10T00:55:01.524 INFO:journalctl@ceph.osd.6.smithi170.stdout:Mar 10 00:55:01 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-osd-6[43049]: 2024-03-10T00:55:01.300+0000 7f5852423700 -1 osd.6 71 heartbeat_check: no reply from 172.21.15.106:6814 osd.1 since back 2024-03-10T00:54:28.453247+0000 front 2024-03-10T00:54:28.453219+0000 (oldest deadline 2024-03-10T00:54:53.753144+0000) 2024-03-10T00:55:01.524 INFO:journalctl@ceph.osd.7.smithi170.stdout:Mar 10 00:55:01 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-osd-7[47108]: 2024-03-10T00:55:01.070+0000 7f7b81779700 -1 osd.7 71 heartbeat_check: no reply from 172.21.15.106:6806 osd.0 since back 2024-03-10T00:54:19.432267+0000 front 2024-03-10T00:54:19.432106+0000 (oldest deadline 2024-03-10T00:54:43.532104+0000) 2024-03-10T00:55:01.524 INFO:journalctl@ceph.osd.7.smithi170.stdout:Mar 10 00:55:01 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-osd-7[47108]: 2024-03-10T00:55:01.070+0000 7f7b81779700 -1 osd.7 71 heartbeat_check: no reply from 172.21.15.106:6814 osd.1 since back 2024-03-10T00:54:30.533390+0000 front 2024-03-10T00:54:30.533458+0000 (oldest deadline 2024-03-10T00:54:54.633272+0000) 2024-03-10T00:55:02.024 INFO:journalctl@ceph.osd.5.smithi170.stdout:Mar 10 00:55:01 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-osd-5[38986]: 2024-03-10T00:55:01.583+0000 7f72444e9700 -1 osd.5 71 heartbeat_check: no reply from 172.21.15.106:6806 osd.0 since back 2024-03-10T00:54:18.209644+0000 front 2024-03-10T00:54:18.209502+0000 (oldest deadline 2024-03-10T00:54:48.210011+0000) 2024-03-10T00:55:02.024 INFO:journalctl@ceph.osd.5.smithi170.stdout:Mar 10 00:55:01 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-osd-5[38986]: 2024-03-10T00:55:01.583+0000 7f72444e9700 -1 osd.5 71 heartbeat_check: no reply from 172.21.15.106:6814 osd.1 since back 2024-03-10T00:54:28.210388+0000 front 2024-03-10T00:54:28.210493+0000 (oldest deadline 2024-03-10T00:54:52.910325+0000) 2024-03-10T00:55:02.524 INFO:journalctl@ceph.osd.6.smithi170.stdout:Mar 10 00:55:02 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-osd-6[43049]: 2024-03-10T00:55:02.302+0000 7f5852423700 -1 osd.6 71 heartbeat_check: no reply from 172.21.15.106:6806 osd.0 since back 2024-03-10T00:54:17.852283+0000 front 2024-03-10T00:54:17.852086+0000 (oldest deadline 2024-03-10T00:54:43.752263+0000) 2024-03-10T00:55:02.524 INFO:journalctl@ceph.osd.6.smithi170.stdout:Mar 10 00:55:02 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-osd-6[43049]: 2024-03-10T00:55:02.302+0000 7f5852423700 -1 osd.6 71 heartbeat_check: no reply from 172.21.15.106:6814 osd.1 since back 2024-03-10T00:54:28.453247+0000 front 2024-03-10T00:54:28.453219+0000 (oldest deadline 2024-03-10T00:54:53.753144+0000) 2024-03-10T00:55:02.524 INFO:journalctl@ceph.osd.7.smithi170.stdout:Mar 10 00:55:02 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-osd-7[47108]: 2024-03-10T00:55:02.111+0000 7f7b81779700 -1 osd.7 71 heartbeat_check: no reply from 172.21.15.106:6806 osd.0 since back 2024-03-10T00:54:19.432267+0000 front 2024-03-10T00:54:19.432106+0000 (oldest deadline 2024-03-10T00:54:43.532104+0000) 2024-03-10T00:55:02.524 INFO:journalctl@ceph.osd.7.smithi170.stdout:Mar 10 00:55:02 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-osd-7[47108]: 2024-03-10T00:55:02.111+0000 7f7b81779700 -1 osd.7 71 heartbeat_check: no reply from 172.21.15.106:6814 osd.1 since back 2024-03-10T00:54:30.533390+0000 front 2024-03-10T00:54:30.533458+0000 (oldest deadline 2024-03-10T00:54:54.633272+0000) 2024-03-10T00:55:02.524 INFO:journalctl@ceph.osd.7.smithi170.stdout:Mar 10 00:55:02 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-osd-7[47108]: 2024-03-10T00:55:02.111+0000 7f7b81779700 -1 osd.7 71 heartbeat_check: no reply from 172.21.15.106:6822 osd.2 since back 2024-03-10T00:54:38.435593+0000 front 2024-03-10T00:54:38.435565+0000 (oldest deadline 2024-03-10T00:55:01.935552+0000) 2024-03-10T00:55:03.024 INFO:journalctl@ceph.osd.5.smithi170.stdout:Mar 10 00:55:02 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-osd-5[38986]: 2024-03-10T00:55:02.576+0000 7f72444e9700 -1 osd.5 71 heartbeat_check: no reply from 172.21.15.106:6806 osd.0 since back 2024-03-10T00:54:18.209644+0000 front 2024-03-10T00:54:18.209502+0000 (oldest deadline 2024-03-10T00:54:48.210011+0000) 2024-03-10T00:55:03.024 INFO:journalctl@ceph.osd.5.smithi170.stdout:Mar 10 00:55:02 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-osd-5[38986]: 2024-03-10T00:55:02.576+0000 7f72444e9700 -1 osd.5 71 heartbeat_check: no reply from 172.21.15.106:6814 osd.1 since back 2024-03-10T00:54:28.210388+0000 front 2024-03-10T00:54:28.210493+0000 (oldest deadline 2024-03-10T00:54:52.910325+0000) 2024-03-10T00:55:03.524 INFO:journalctl@ceph.osd.6.smithi170.stdout:Mar 10 00:55:03 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-osd-6[43049]: 2024-03-10T00:55:03.283+0000 7f5852423700 -1 osd.6 71 heartbeat_check: no reply from 172.21.15.106:6806 osd.0 since back 2024-03-10T00:54:17.852283+0000 front 2024-03-10T00:54:17.852086+0000 (oldest deadline 2024-03-10T00:54:43.752263+0000) 2024-03-10T00:55:03.524 INFO:journalctl@ceph.osd.6.smithi170.stdout:Mar 10 00:55:03 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-osd-6[43049]: 2024-03-10T00:55:03.283+0000 7f5852423700 -1 osd.6 71 heartbeat_check: no reply from 172.21.15.106:6814 osd.1 since back 2024-03-10T00:54:28.453247+0000 front 2024-03-10T00:54:28.453219+0000 (oldest deadline 2024-03-10T00:54:53.753144+0000) 2024-03-10T00:55:03.524 INFO:journalctl@ceph.osd.7.smithi170.stdout:Mar 10 00:55:03 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-osd-7[47108]: 2024-03-10T00:55:03.143+0000 7f7b81779700 -1 osd.7 71 heartbeat_check: no reply from 172.21.15.106:6806 osd.0 since back 2024-03-10T00:54:19.432267+0000 front 2024-03-10T00:54:19.432106+0000 (oldest deadline 2024-03-10T00:54:43.532104+0000) 2024-03-10T00:55:03.524 INFO:journalctl@ceph.osd.7.smithi170.stdout:Mar 10 00:55:03 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-osd-7[47108]: 2024-03-10T00:55:03.143+0000 7f7b81779700 -1 osd.7 71 heartbeat_check: no reply from 172.21.15.106:6814 osd.1 since back 2024-03-10T00:54:30.533390+0000 front 2024-03-10T00:54:30.533458+0000 (oldest deadline 2024-03-10T00:54:54.633272+0000) 2024-03-10T00:55:03.524 INFO:journalctl@ceph.osd.7.smithi170.stdout:Mar 10 00:55:03 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-osd-7[47108]: 2024-03-10T00:55:03.143+0000 7f7b81779700 -1 osd.7 71 heartbeat_check: no reply from 172.21.15.106:6822 osd.2 since back 2024-03-10T00:54:38.435593+0000 front 2024-03-10T00:54:38.435565+0000 (oldest deadline 2024-03-10T00:55:01.935552+0000) 2024-03-10T00:55:04.024 INFO:journalctl@ceph.osd.5.smithi170.stdout:Mar 10 00:55:03 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-osd-5[38986]: 2024-03-10T00:55:03.613+0000 7f72444e9700 -1 osd.5 71 heartbeat_check: no reply from 172.21.15.106:6806 osd.0 since back 2024-03-10T00:54:18.209644+0000 front 2024-03-10T00:54:18.209502+0000 (oldest deadline 2024-03-10T00:54:48.210011+0000) 2024-03-10T00:55:04.024 INFO:journalctl@ceph.osd.5.smithi170.stdout:Mar 10 00:55:03 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-osd-5[38986]: 2024-03-10T00:55:03.613+0000 7f72444e9700 -1 osd.5 71 heartbeat_check: no reply from 172.21.15.106:6814 osd.1 since back 2024-03-10T00:54:28.210388+0000 front 2024-03-10T00:54:28.210493+0000 (oldest deadline 2024-03-10T00:54:52.910325+0000) 2024-03-10T00:55:04.024 INFO:journalctl@ceph.osd.5.smithi170.stdout:Mar 10 00:55:03 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-osd-5[38986]: 2024-03-10T00:55:03.613+0000 7f72444e9700 -1 osd.5 71 heartbeat_check: no reply from 172.21.15.106:6822 osd.2 since back 2024-03-10T00:54:36.411399+0000 front 2024-03-10T00:54:36.411367+0000 (oldest deadline 2024-03-10T00:55:03.411817+0000) 2024-03-10T00:55:04.524 INFO:journalctl@ceph.osd.7.smithi170.stdout:Mar 10 00:55:04 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-osd-7[47108]: 2024-03-10T00:55:04.132+0000 7f7b81779700 -1 osd.7 71 heartbeat_check: no reply from 172.21.15.106:6806 osd.0 since back 2024-03-10T00:54:19.432267+0000 front 2024-03-10T00:54:19.432106+0000 (oldest deadline 2024-03-10T00:54:43.532104+0000) 2024-03-10T00:55:04.524 INFO:journalctl@ceph.osd.7.smithi170.stdout:Mar 10 00:55:04 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-osd-7[47108]: 2024-03-10T00:55:04.133+0000 7f7b81779700 -1 osd.7 71 heartbeat_check: no reply from 172.21.15.106:6814 osd.1 since back 2024-03-10T00:54:30.533390+0000 front 2024-03-10T00:54:30.533458+0000 (oldest deadline 2024-03-10T00:54:54.633272+0000) 2024-03-10T00:55:04.524 INFO:journalctl@ceph.osd.7.smithi170.stdout:Mar 10 00:55:04 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-osd-7[47108]: 2024-03-10T00:55:04.133+0000 7f7b81779700 -1 osd.7 71 heartbeat_check: no reply from 172.21.15.106:6822 osd.2 since back 2024-03-10T00:54:38.435593+0000 front 2024-03-10T00:54:38.435565+0000 (oldest deadline 2024-03-10T00:55:01.935552+0000) 2024-03-10T00:55:04.524 INFO:journalctl@ceph.osd.7.smithi170.stdout:Mar 10 00:55:04 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-osd-7[47108]: 2024-03-10T00:55:04.133+0000 7f7b81779700 -1 osd.7 71 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.24440.0:149 2.0 2:353ec8f9:::gateway.conf:head [getxattr epoch in=5b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e71) 2024-03-10T00:55:04.524 INFO:journalctl@ceph.osd.6.smithi170.stdout:Mar 10 00:55:04 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-osd-6[43049]: 2024-03-10T00:55:04.312+0000 7f5852423700 -1 osd.6 71 heartbeat_check: no reply from 172.21.15.106:6806 osd.0 since back 2024-03-10T00:54:17.852283+0000 front 2024-03-10T00:54:17.852086+0000 (oldest deadline 2024-03-10T00:54:43.752263+0000) 2024-03-10T00:55:04.524 INFO:journalctl@ceph.osd.6.smithi170.stdout:Mar 10 00:55:04 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-osd-6[43049]: 2024-03-10T00:55:04.312+0000 7f5852423700 -1 osd.6 71 heartbeat_check: no reply from 172.21.15.106:6814 osd.1 since back 2024-03-10T00:54:28.453247+0000 front 2024-03-10T00:54:28.453219+0000 (oldest deadline 2024-03-10T00:54:53.753144+0000) 2024-03-10T00:55:04.524 INFO:journalctl@ceph.osd.6.smithi170.stdout:Mar 10 00:55:04 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-osd-6[43049]: 2024-03-10T00:55:04.312+0000 7f5852423700 -1 osd.6 71 heartbeat_check: no reply from 172.21.15.106:6822 osd.2 since back 2024-03-10T00:54:38.954861+0000 front 2024-03-10T00:54:38.954901+0000 (oldest deadline 2024-03-10T00:55:03.654750+0000) 2024-03-10T00:55:05.015 INFO:journalctl@ceph.osd.5.smithi170.stdout:Mar 10 00:55:04 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-osd-5[38986]: 2024-03-10T00:55:04.660+0000 7f72444e9700 -1 osd.5 71 heartbeat_check: no reply from 172.21.15.106:6806 osd.0 since back 2024-03-10T00:54:18.209644+0000 front 2024-03-10T00:54:18.209502+0000 (oldest deadline 2024-03-10T00:54:48.210011+0000) 2024-03-10T00:55:05.015 INFO:journalctl@ceph.osd.5.smithi170.stdout:Mar 10 00:55:04 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-osd-5[38986]: 2024-03-10T00:55:04.660+0000 7f72444e9700 -1 osd.5 71 heartbeat_check: no reply from 172.21.15.106:6814 osd.1 since back 2024-03-10T00:54:28.210388+0000 front 2024-03-10T00:54:28.210493+0000 (oldest deadline 2024-03-10T00:54:52.910325+0000) 2024-03-10T00:55:05.015 INFO:journalctl@ceph.osd.5.smithi170.stdout:Mar 10 00:55:04 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-osd-5[38986]: 2024-03-10T00:55:04.660+0000 7f72444e9700 -1 osd.5 71 heartbeat_check: no reply from 172.21.15.106:6822 osd.2 since back 2024-03-10T00:54:36.411399+0000 front 2024-03-10T00:54:36.411367+0000 (oldest deadline 2024-03-10T00:55:03.411817+0000) 2024-03-10T00:55:05.274 INFO:journalctl@ceph.osd.7.smithi170.stdout:Mar 10 00:55:05 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-osd-7[47108]: 2024-03-10T00:55:05.131+0000 7f7b81779700 -1 osd.7 71 heartbeat_check: no reply from 172.21.15.106:6806 osd.0 since back 2024-03-10T00:54:19.432267+0000 front 2024-03-10T00:54:19.432106+0000 (oldest deadline 2024-03-10T00:54:43.532104+0000) 2024-03-10T00:55:05.274 INFO:journalctl@ceph.osd.7.smithi170.stdout:Mar 10 00:55:05 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-osd-7[47108]: 2024-03-10T00:55:05.131+0000 7f7b81779700 -1 osd.7 71 heartbeat_check: no reply from 172.21.15.106:6814 osd.1 since back 2024-03-10T00:54:30.533390+0000 front 2024-03-10T00:54:30.533458+0000 (oldest deadline 2024-03-10T00:54:54.633272+0000) 2024-03-10T00:55:05.274 INFO:journalctl@ceph.osd.7.smithi170.stdout:Mar 10 00:55:05 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-osd-7[47108]: 2024-03-10T00:55:05.131+0000 7f7b81779700 -1 osd.7 71 heartbeat_check: no reply from 172.21.15.106:6822 osd.2 since back 2024-03-10T00:54:38.435593+0000 front 2024-03-10T00:54:38.435565+0000 (oldest deadline 2024-03-10T00:55:01.935552+0000) 2024-03-10T00:55:05.274 INFO:journalctl@ceph.osd.7.smithi170.stdout:Mar 10 00:55:05 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-osd-7[47108]: 2024-03-10T00:55:05.131+0000 7f7b81779700 -1 osd.7 71 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.24440.0:149 2.0 2:353ec8f9:::gateway.conf:head [getxattr epoch in=5b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e71) 2024-03-10T00:55:05.759 INFO:journalctl@ceph.osd.6.smithi170.stdout:Mar 10 00:55:05 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-osd-6[43049]: 2024-03-10T00:55:05.303+0000 7f5852423700 -1 osd.6 71 heartbeat_check: no reply from 172.21.15.106:6806 osd.0 since back 2024-03-10T00:54:17.852283+0000 front 2024-03-10T00:54:17.852086+0000 (oldest deadline 2024-03-10T00:54:43.752263+0000) 2024-03-10T00:55:05.759 INFO:journalctl@ceph.osd.6.smithi170.stdout:Mar 10 00:55:05 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-osd-6[43049]: 2024-03-10T00:55:05.303+0000 7f5852423700 -1 osd.6 71 heartbeat_check: no reply from 172.21.15.106:6814 osd.1 since back 2024-03-10T00:54:28.453247+0000 front 2024-03-10T00:54:28.453219+0000 (oldest deadline 2024-03-10T00:54:53.753144+0000) 2024-03-10T00:55:05.759 INFO:journalctl@ceph.osd.6.smithi170.stdout:Mar 10 00:55:05 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-osd-6[43049]: 2024-03-10T00:55:05.303+0000 7f5852423700 -1 osd.6 71 heartbeat_check: no reply from 172.21.15.106:6822 osd.2 since back 2024-03-10T00:54:38.954861+0000 front 2024-03-10T00:54:38.954901+0000 (oldest deadline 2024-03-10T00:55:03.654750+0000) 2024-03-10T00:55:06.024 INFO:journalctl@ceph.osd.5.smithi170.stdout:Mar 10 00:55:05 smithi170 podman[66372]: 2024-03-10 00:55:05.762387891 +0000 UTC m=+5.558467768 container died 6993eeb86c16faf28cf9f9fdd31b101a3c306445c1091cddce29c8399b894688 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bc4a24690ce5d7255d490e21058d8c114b55e1b93a3d8345f90b6700ad29dada, name=ceph-75c37498-de77-11ee-95c7-87774f69a715-osd-5, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, ceph=True, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, RELEASE=wip-yuri4-testing-2024-03-05-0854-831bc18, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240222) 2024-03-10T00:55:06.314 INFO:journalctl@ceph.osd.5.smithi170.stdout:Mar 10 00:55:06 smithi170 podman[66372]: 2024-03-10 00:55:06.207786308 +0000 UTC m=+6.003866183 container cleanup 6993eeb86c16faf28cf9f9fdd31b101a3c306445c1091cddce29c8399b894688 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bc4a24690ce5d7255d490e21058d8c114b55e1b93a3d8345f90b6700ad29dada, name=ceph-75c37498-de77-11ee-95c7-87774f69a715-osd-5, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240222, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-yuri4-testing-2024-03-05-0854-831bc18, ceph=True, maintainer=Guillaume Abrioux ) 2024-03-10T00:55:06.315 INFO:journalctl@ceph.osd.5.smithi170.stdout:Mar 10 00:55:06 smithi170 bash[66372]: ceph-75c37498-de77-11ee-95c7-87774f69a715-osd-5 2024-03-10T00:55:06.315 INFO:journalctl@ceph.osd.7.smithi170.stdout:Mar 10 00:55:06 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-osd-7[47108]: 2024-03-10T00:55:06.104+0000 7f7b81779700 -1 osd.7 71 heartbeat_check: no reply from 172.21.15.106:6806 osd.0 since back 2024-03-10T00:54:19.432267+0000 front 2024-03-10T00:54:19.432106+0000 (oldest deadline 2024-03-10T00:54:43.532104+0000) 2024-03-10T00:55:06.315 INFO:journalctl@ceph.osd.7.smithi170.stdout:Mar 10 00:55:06 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-osd-7[47108]: 2024-03-10T00:55:06.104+0000 7f7b81779700 -1 osd.7 71 heartbeat_check: no reply from 172.21.15.106:6814 osd.1 since back 2024-03-10T00:54:30.533390+0000 front 2024-03-10T00:54:30.533458+0000 (oldest deadline 2024-03-10T00:54:54.633272+0000) 2024-03-10T00:55:06.315 INFO:journalctl@ceph.osd.7.smithi170.stdout:Mar 10 00:55:06 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-osd-7[47108]: 2024-03-10T00:55:06.104+0000 7f7b81779700 -1 osd.7 71 heartbeat_check: no reply from 172.21.15.106:6822 osd.2 since back 2024-03-10T00:54:38.435593+0000 front 2024-03-10T00:54:38.435565+0000 (oldest deadline 2024-03-10T00:55:01.935552+0000) 2024-03-10T00:55:06.315 INFO:journalctl@ceph.osd.7.smithi170.stdout:Mar 10 00:55:06 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-osd-7[47108]: 2024-03-10T00:55:06.104+0000 7f7b81779700 -1 osd.7 71 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.24440.0:149 2.0 2:353ec8f9:::gateway.conf:head [getxattr epoch in=5b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e71) 2024-03-10T00:55:06.774 INFO:journalctl@ceph.osd.5.smithi170.stdout:Mar 10 00:55:06 smithi170 podman[66577]: 2024-03-10 00:55:06.375862526 +0000 UTC m=+0.606787431 container remove 6993eeb86c16faf28cf9f9fdd31b101a3c306445c1091cddce29c8399b894688 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bc4a24690ce5d7255d490e21058d8c114b55e1b93a3d8345f90b6700ad29dada, name=ceph-75c37498-de77-11ee-95c7-87774f69a715-osd-5, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240222, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_CLEAN=True, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, RELEASE=wip-yuri4-testing-2024-03-05-0854-831bc18, GIT_BRANCH=HEAD, ceph=True, org.label-schema.license=GPLv2) 2024-03-10T00:55:06.774 INFO:journalctl@ceph.osd.6.smithi170.stdout:Mar 10 00:55:06 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-osd-6[43049]: 2024-03-10T00:55:06.316+0000 7f5852423700 -1 osd.6 71 heartbeat_check: no reply from 172.21.15.106:6806 osd.0 since back 2024-03-10T00:54:17.852283+0000 front 2024-03-10T00:54:17.852086+0000 (oldest deadline 2024-03-10T00:54:43.752263+0000) 2024-03-10T00:55:06.774 INFO:journalctl@ceph.osd.6.smithi170.stdout:Mar 10 00:55:06 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-osd-6[43049]: 2024-03-10T00:55:06.316+0000 7f5852423700 -1 osd.6 71 heartbeat_check: no reply from 172.21.15.106:6814 osd.1 since back 2024-03-10T00:54:28.453247+0000 front 2024-03-10T00:54:28.453219+0000 (oldest deadline 2024-03-10T00:54:53.753144+0000) 2024-03-10T00:55:06.774 INFO:journalctl@ceph.osd.6.smithi170.stdout:Mar 10 00:55:06 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-osd-6[43049]: 2024-03-10T00:55:06.316+0000 7f5852423700 -1 osd.6 71 heartbeat_check: no reply from 172.21.15.106:6822 osd.2 since back 2024-03-10T00:54:38.954861+0000 front 2024-03-10T00:54:38.954901+0000 (oldest deadline 2024-03-10T00:55:03.654750+0000) 2024-03-10T00:55:07.111 INFO:journalctl@ceph.osd.5.smithi170.stdout:Mar 10 00:55:06 smithi170 podman[66738]: 2024-03-10 00:55:06.756774097 +0000 UTC m=+0.022640890 image pull 01a4f6a6bb3e92ab9ab39a2f835684ea9b10e6ec9fb81ac08ebea4165583a538 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bc4a24690ce5d7255d490e21058d8c114b55e1b93a3d8345f90b6700ad29dada 2024-03-10T00:55:07.111 INFO:journalctl@ceph.osd.5.smithi170.stdout:Mar 10 00:55:06 smithi170 podman[66738]: 2024-03-10 00:55:06.885050541 +0000 UTC m=+0.150917362 container create 0b952146013bb203cf1c42d7b35eb05d2c7a15213438e8593ba2f8cfbfa4b928 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bc4a24690ce5d7255d490e21058d8c114b55e1b93a3d8345f90b6700ad29dada, name=ceph-75c37498-de77-11ee-95c7-87774f69a715-osd-5-deactivate, GIT_CLEAN=True, RELEASE=wip-yuri4-testing-2024-03-05-0854-831bc18, io.buildah.version=1.33.5, org.label-schema.build-date=20240222, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=) 2024-03-10T00:55:07.111 INFO:journalctl@ceph.osd.5.smithi170.stdout:Mar 10 00:55:07 smithi170 podman[66738]: 2024-03-10 00:55:07.113769608 +0000 UTC m=+0.379636385 container init 0b952146013bb203cf1c42d7b35eb05d2c7a15213438e8593ba2f8cfbfa4b928 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bc4a24690ce5d7255d490e21058d8c114b55e1b93a3d8345f90b6700ad29dada, name=ceph-75c37498-de77-11ee-95c7-87774f69a715-osd-5-deactivate, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240222, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-yuri4-testing-2024-03-05-0854-831bc18, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, ceph=True, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, org.label-schema.vendor=CentOS) 2024-03-10T00:55:07.111 INFO:journalctl@ceph.osd.7.smithi170.stdout:Mar 10 00:55:07 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-osd-7[47108]: 2024-03-10T00:55:07.096+0000 7f7b81779700 -1 osd.7 71 heartbeat_check: no reply from 172.21.15.106:6806 osd.0 since back 2024-03-10T00:54:19.432267+0000 front 2024-03-10T00:54:19.432106+0000 (oldest deadline 2024-03-10T00:54:43.532104+0000) 2024-03-10T00:55:07.111 INFO:journalctl@ceph.osd.7.smithi170.stdout:Mar 10 00:55:07 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-osd-7[47108]: 2024-03-10T00:55:07.096+0000 7f7b81779700 -1 osd.7 71 heartbeat_check: no reply from 172.21.15.106:6814 osd.1 since back 2024-03-10T00:54:30.533390+0000 front 2024-03-10T00:54:30.533458+0000 (oldest deadline 2024-03-10T00:54:54.633272+0000) 2024-03-10T00:55:07.111 INFO:journalctl@ceph.osd.7.smithi170.stdout:Mar 10 00:55:07 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-osd-7[47108]: 2024-03-10T00:55:07.096+0000 7f7b81779700 -1 osd.7 71 heartbeat_check: no reply from 172.21.15.106:6822 osd.2 since back 2024-03-10T00:54:38.435593+0000 front 2024-03-10T00:54:38.435565+0000 (oldest deadline 2024-03-10T00:55:01.935552+0000) 2024-03-10T00:55:07.111 INFO:journalctl@ceph.osd.7.smithi170.stdout:Mar 10 00:55:07 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-osd-7[47108]: 2024-03-10T00:55:07.096+0000 7f7b81779700 -1 osd.7 71 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.24440.0:149 2.0 2:353ec8f9:::gateway.conf:head [getxattr epoch in=5b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e71) 2024-03-10T00:55:07.363 INFO:journalctl@ceph.osd.5.smithi170.stdout:Mar 10 00:55:07 smithi170 podman[66738]: 2024-03-10 00:55:07.119667174 +0000 UTC m=+0.385533952 container start 0b952146013bb203cf1c42d7b35eb05d2c7a15213438e8593ba2f8cfbfa4b928 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bc4a24690ce5d7255d490e21058d8c114b55e1b93a3d8345f90b6700ad29dada, name=ceph-75c37498-de77-11ee-95c7-87774f69a715-osd-5-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, org.label-schema.build-date=20240222, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_CLEAN=True, ceph=True, RELEASE=wip-yuri4-testing-2024-03-05-0854-831bc18, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-10T00:55:07.363 INFO:journalctl@ceph.osd.5.smithi170.stdout:Mar 10 00:55:07 smithi170 podman[66738]: 2024-03-10 00:55:07.221015558 +0000 UTC m=+0.486882336 container attach 0b952146013bb203cf1c42d7b35eb05d2c7a15213438e8593ba2f8cfbfa4b928 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bc4a24690ce5d7255d490e21058d8c114b55e1b93a3d8345f90b6700ad29dada, name=ceph-75c37498-de77-11ee-95c7-87774f69a715-osd-5-deactivate, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.build-date=20240222, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-yuri4-testing-2024-03-05-0854-831bc18, GIT_BRANCH=HEAD) 2024-03-10T00:55:07.774 INFO:journalctl@ceph.osd.5.smithi170.stdout:Mar 10 00:55:07 smithi170 podman[66738]: 2024-03-10 00:55:07.453733229 +0000 UTC m=+0.719600012 container died 0b952146013bb203cf1c42d7b35eb05d2c7a15213438e8593ba2f8cfbfa4b928 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bc4a24690ce5d7255d490e21058d8c114b55e1b93a3d8345f90b6700ad29dada, name=ceph-75c37498-de77-11ee-95c7-87774f69a715-osd-5-deactivate, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, RELEASE=wip-yuri4-testing-2024-03-05-0854-831bc18, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240222, ceph=True, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, maintainer=Guillaume Abrioux ) 2024-03-10T00:55:07.774 INFO:journalctl@ceph.osd.6.smithi170.stdout:Mar 10 00:55:07 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-osd-6[43049]: 2024-03-10T00:55:07.364+0000 7f5852423700 -1 osd.6 71 heartbeat_check: no reply from 172.21.15.106:6806 osd.0 since back 2024-03-10T00:54:17.852283+0000 front 2024-03-10T00:54:17.852086+0000 (oldest deadline 2024-03-10T00:54:43.752263+0000) 2024-03-10T00:55:07.774 INFO:journalctl@ceph.osd.6.smithi170.stdout:Mar 10 00:55:07 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-osd-6[43049]: 2024-03-10T00:55:07.364+0000 7f5852423700 -1 osd.6 71 heartbeat_check: no reply from 172.21.15.106:6814 osd.1 since back 2024-03-10T00:54:28.453247+0000 front 2024-03-10T00:54:28.453219+0000 (oldest deadline 2024-03-10T00:54:53.753144+0000) 2024-03-10T00:55:07.774 INFO:journalctl@ceph.osd.6.smithi170.stdout:Mar 10 00:55:07 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-osd-6[43049]: 2024-03-10T00:55:07.364+0000 7f5852423700 -1 osd.6 71 heartbeat_check: no reply from 172.21.15.106:6822 osd.2 since back 2024-03-10T00:54:38.954861+0000 front 2024-03-10T00:54:38.954901+0000 (oldest deadline 2024-03-10T00:55:03.654750+0000) 2024-03-10T00:55:08.274 INFO:journalctl@ceph.osd.7.smithi170.stdout:Mar 10 00:55:08 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-osd-7[47108]: 2024-03-10T00:55:08.094+0000 7f7b81779700 -1 osd.7 71 heartbeat_check: no reply from 172.21.15.106:6806 osd.0 since back 2024-03-10T00:54:19.432267+0000 front 2024-03-10T00:54:19.432106+0000 (oldest deadline 2024-03-10T00:54:43.532104+0000) 2024-03-10T00:55:08.274 INFO:journalctl@ceph.osd.7.smithi170.stdout:Mar 10 00:55:08 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-osd-7[47108]: 2024-03-10T00:55:08.094+0000 7f7b81779700 -1 osd.7 71 heartbeat_check: no reply from 172.21.15.106:6814 osd.1 since back 2024-03-10T00:54:30.533390+0000 front 2024-03-10T00:54:30.533458+0000 (oldest deadline 2024-03-10T00:54:54.633272+0000) 2024-03-10T00:55:08.274 INFO:journalctl@ceph.osd.7.smithi170.stdout:Mar 10 00:55:08 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-osd-7[47108]: 2024-03-10T00:55:08.094+0000 7f7b81779700 -1 osd.7 71 heartbeat_check: no reply from 172.21.15.106:6822 osd.2 since back 2024-03-10T00:54:38.435593+0000 front 2024-03-10T00:54:38.435565+0000 (oldest deadline 2024-03-10T00:55:01.935552+0000) 2024-03-10T00:55:08.274 INFO:journalctl@ceph.osd.7.smithi170.stdout:Mar 10 00:55:08 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-osd-7[47108]: 2024-03-10T00:55:08.094+0000 7f7b81779700 -1 osd.7 71 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.24440.0:149 2.0 2:353ec8f9:::gateway.conf:head [getxattr epoch in=5b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e71) 2024-03-10T00:55:08.321 DEBUG:teuthology.orchestra.run.smithi170:> sudo pkill -f 'journalctl -f -n 0 -u ceph-75c37498-de77-11ee-95c7-87774f69a715@osd.5.service' 2024-03-10T00:55:08.363 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-10T00:55:08.363 INFO:tasks.cephadm.osd.5:Stopped osd.5 2024-03-10T00:55:08.364 INFO:tasks.cephadm.osd.6:Stopping osd.6... 2024-03-10T00:55:08.364 DEBUG:teuthology.orchestra.run.smithi170:> sudo systemctl stop ceph-75c37498-de77-11ee-95c7-87774f69a715@osd.6 2024-03-10T00:55:08.591 INFO:journalctl@ceph.osd.6.smithi170.stdout:Mar 10 00:55:08 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-osd-6[43049]: 2024-03-10T00:55:08.358+0000 7f5852423700 -1 osd.6 71 heartbeat_check: no reply from 172.21.15.106:6806 osd.0 since back 2024-03-10T00:54:17.852283+0000 front 2024-03-10T00:54:17.852086+0000 (oldest deadline 2024-03-10T00:54:43.752263+0000) 2024-03-10T00:55:08.592 INFO:journalctl@ceph.osd.6.smithi170.stdout:Mar 10 00:55:08 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-osd-6[43049]: 2024-03-10T00:55:08.358+0000 7f5852423700 -1 osd.6 71 heartbeat_check: no reply from 172.21.15.106:6814 osd.1 since back 2024-03-10T00:54:28.453247+0000 front 2024-03-10T00:54:28.453219+0000 (oldest deadline 2024-03-10T00:54:53.753144+0000) 2024-03-10T00:55:08.592 INFO:journalctl@ceph.osd.6.smithi170.stdout:Mar 10 00:55:08 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-osd-6[43049]: 2024-03-10T00:55:08.358+0000 7f5852423700 -1 osd.6 71 heartbeat_check: no reply from 172.21.15.106:6822 osd.2 since back 2024-03-10T00:54:38.954861+0000 front 2024-03-10T00:54:38.954901+0000 (oldest deadline 2024-03-10T00:55:03.654750+0000) 2024-03-10T00:55:08.592 INFO:journalctl@ceph.osd.6.smithi170.stdout:Mar 10 00:55:08 smithi170 systemd[1]: Stopping Ceph osd.6 for 75c37498-de77-11ee-95c7-87774f69a715... 2024-03-10T00:55:09.274 INFO:journalctl@ceph.osd.6.smithi170.stdout:Mar 10 00:55:08 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-osd-6[43049]: 2024-03-10T00:55:08.975+0000 7f5855e2f700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.6 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-10T00:55:09.274 INFO:journalctl@ceph.osd.6.smithi170.stdout:Mar 10 00:55:08 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-osd-6[43049]: 2024-03-10T00:55:08.975+0000 7f5855e2f700 -1 osd.6 71 *** Got signal Terminated *** 2024-03-10T00:55:09.274 INFO:journalctl@ceph.osd.6.smithi170.stdout:Mar 10 00:55:08 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-osd-6[43049]: 2024-03-10T00:55:08.975+0000 7f5855e2f700 -1 osd.6 71 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-10T00:55:09.274 INFO:journalctl@ceph.osd.7.smithi170.stdout:Mar 10 00:55:09 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-osd-7[47108]: 2024-03-10T00:55:09.052+0000 7f7b81779700 -1 osd.7 71 heartbeat_check: no reply from 172.21.15.106:6806 osd.0 since back 2024-03-10T00:54:19.432267+0000 front 2024-03-10T00:54:19.432106+0000 (oldest deadline 2024-03-10T00:54:43.532104+0000) 2024-03-10T00:55:09.274 INFO:journalctl@ceph.osd.7.smithi170.stdout:Mar 10 00:55:09 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-osd-7[47108]: 2024-03-10T00:55:09.052+0000 7f7b81779700 -1 osd.7 71 heartbeat_check: no reply from 172.21.15.106:6814 osd.1 since back 2024-03-10T00:54:30.533390+0000 front 2024-03-10T00:54:30.533458+0000 (oldest deadline 2024-03-10T00:54:54.633272+0000) 2024-03-10T00:55:09.274 INFO:journalctl@ceph.osd.7.smithi170.stdout:Mar 10 00:55:09 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-osd-7[47108]: 2024-03-10T00:55:09.052+0000 7f7b81779700 -1 osd.7 71 heartbeat_check: no reply from 172.21.15.106:6822 osd.2 since back 2024-03-10T00:54:38.435593+0000 front 2024-03-10T00:54:38.435565+0000 (oldest deadline 2024-03-10T00:55:01.935552+0000) 2024-03-10T00:55:09.274 INFO:journalctl@ceph.osd.7.smithi170.stdout:Mar 10 00:55:09 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-osd-7[47108]: 2024-03-10T00:55:09.052+0000 7f7b81779700 -1 osd.7 71 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.24440.0:149 2.0 2:353ec8f9:::gateway.conf:head [getxattr epoch in=5b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e71) 2024-03-10T00:55:09.774 INFO:journalctl@ceph.osd.6.smithi170.stdout:Mar 10 00:55:09 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-osd-6[43049]: 2024-03-10T00:55:09.331+0000 7f5852423700 -1 osd.6 71 heartbeat_check: no reply from 172.21.15.106:6806 osd.0 since back 2024-03-10T00:54:17.852283+0000 front 2024-03-10T00:54:17.852086+0000 (oldest deadline 2024-03-10T00:54:43.752263+0000) 2024-03-10T00:55:09.774 INFO:journalctl@ceph.osd.6.smithi170.stdout:Mar 10 00:55:09 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-osd-6[43049]: 2024-03-10T00:55:09.331+0000 7f5852423700 -1 osd.6 71 heartbeat_check: no reply from 172.21.15.106:6814 osd.1 since back 2024-03-10T00:54:28.453247+0000 front 2024-03-10T00:54:28.453219+0000 (oldest deadline 2024-03-10T00:54:53.753144+0000) 2024-03-10T00:55:09.774 INFO:journalctl@ceph.osd.6.smithi170.stdout:Mar 10 00:55:09 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-osd-6[43049]: 2024-03-10T00:55:09.331+0000 7f5852423700 -1 osd.6 71 heartbeat_check: no reply from 172.21.15.106:6822 osd.2 since back 2024-03-10T00:54:38.954861+0000 front 2024-03-10T00:54:38.954901+0000 (oldest deadline 2024-03-10T00:55:03.654750+0000) 2024-03-10T00:55:10.173 INFO:journalctl@ceph.osd.7.smithi170.stdout:Mar 10 00:55:10 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-osd-7[47108]: 2024-03-10T00:55:10.005+0000 7f7b81779700 -1 osd.7 71 heartbeat_check: no reply from 172.21.15.106:6806 osd.0 since back 2024-03-10T00:54:19.432267+0000 front 2024-03-10T00:54:19.432106+0000 (oldest deadline 2024-03-10T00:54:43.532104+0000) 2024-03-10T00:55:10.174 INFO:journalctl@ceph.osd.7.smithi170.stdout:Mar 10 00:55:10 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-osd-7[47108]: 2024-03-10T00:55:10.005+0000 7f7b81779700 -1 osd.7 71 heartbeat_check: no reply from 172.21.15.106:6814 osd.1 since back 2024-03-10T00:54:30.533390+0000 front 2024-03-10T00:54:30.533458+0000 (oldest deadline 2024-03-10T00:54:54.633272+0000) 2024-03-10T00:55:10.174 INFO:journalctl@ceph.osd.7.smithi170.stdout:Mar 10 00:55:10 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-osd-7[47108]: 2024-03-10T00:55:10.005+0000 7f7b81779700 -1 osd.7 71 heartbeat_check: no reply from 172.21.15.106:6822 osd.2 since back 2024-03-10T00:54:38.435593+0000 front 2024-03-10T00:54:38.435565+0000 (oldest deadline 2024-03-10T00:55:01.935552+0000) 2024-03-10T00:55:10.174 INFO:journalctl@ceph.osd.7.smithi170.stdout:Mar 10 00:55:10 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-osd-7[47108]: 2024-03-10T00:55:10.005+0000 7f7b81779700 -1 osd.7 71 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.24440.0:149 2.0 2:353ec8f9:::gateway.conf:head [getxattr epoch in=5b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e71) 2024-03-10T00:55:10.524 INFO:journalctl@ceph.osd.6.smithi170.stdout:Mar 10 00:55:10 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-osd-6[43049]: 2024-03-10T00:55:10.311+0000 7f5852423700 -1 osd.6 71 heartbeat_check: no reply from 172.21.15.106:6806 osd.0 since back 2024-03-10T00:54:17.852283+0000 front 2024-03-10T00:54:17.852086+0000 (oldest deadline 2024-03-10T00:54:43.752263+0000) 2024-03-10T00:55:10.524 INFO:journalctl@ceph.osd.6.smithi170.stdout:Mar 10 00:55:10 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-osd-6[43049]: 2024-03-10T00:55:10.311+0000 7f5852423700 -1 osd.6 71 heartbeat_check: no reply from 172.21.15.106:6814 osd.1 since back 2024-03-10T00:54:28.453247+0000 front 2024-03-10T00:54:28.453219+0000 (oldest deadline 2024-03-10T00:54:53.753144+0000) 2024-03-10T00:55:10.524 INFO:journalctl@ceph.osd.6.smithi170.stdout:Mar 10 00:55:10 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-osd-6[43049]: 2024-03-10T00:55:10.311+0000 7f5852423700 -1 osd.6 71 heartbeat_check: no reply from 172.21.15.106:6822 osd.2 since back 2024-03-10T00:54:38.954861+0000 front 2024-03-10T00:54:38.954901+0000 (oldest deadline 2024-03-10T00:55:03.654750+0000) 2024-03-10T00:55:10.524 INFO:journalctl@ceph.osd.6.smithi170.stdout:Mar 10 00:55:10 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-osd-6[43049]: 2024-03-10T00:55:10.311+0000 7f5852423700 -1 osd.6 71 heartbeat_check: no reply from 172.21.15.106:6830 osd.3 since back 2024-03-10T00:54:45.955671+0000 front 2024-03-10T00:54:45.955712+0000 (oldest deadline 2024-03-10T00:55:10.055578+0000) 2024-03-10T00:55:11.180 INFO:journalctl@ceph.osd.7.smithi170.stdout:Mar 10 00:55:11 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-osd-7[47108]: 2024-03-10T00:55:11.053+0000 7f7b81779700 -1 osd.7 71 heartbeat_check: no reply from 172.21.15.106:6806 osd.0 since back 2024-03-10T00:54:19.432267+0000 front 2024-03-10T00:54:19.432106+0000 (oldest deadline 2024-03-10T00:54:43.532104+0000) 2024-03-10T00:55:11.180 INFO:journalctl@ceph.osd.7.smithi170.stdout:Mar 10 00:55:11 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-osd-7[47108]: 2024-03-10T00:55:11.053+0000 7f7b81779700 -1 osd.7 71 heartbeat_check: no reply from 172.21.15.106:6814 osd.1 since back 2024-03-10T00:54:30.533390+0000 front 2024-03-10T00:54:30.533458+0000 (oldest deadline 2024-03-10T00:54:54.633272+0000) 2024-03-10T00:55:11.180 INFO:journalctl@ceph.osd.7.smithi170.stdout:Mar 10 00:55:11 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-osd-7[47108]: 2024-03-10T00:55:11.053+0000 7f7b81779700 -1 osd.7 71 heartbeat_check: no reply from 172.21.15.106:6822 osd.2 since back 2024-03-10T00:54:38.435593+0000 front 2024-03-10T00:54:38.435565+0000 (oldest deadline 2024-03-10T00:55:01.935552+0000) 2024-03-10T00:55:11.180 INFO:journalctl@ceph.osd.7.smithi170.stdout:Mar 10 00:55:11 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-osd-7[47108]: 2024-03-10T00:55:11.053+0000 7f7b81779700 -1 osd.7 71 heartbeat_check: no reply from 172.21.15.106:6830 osd.3 since back 2024-03-10T00:54:46.636263+0000 front 2024-03-10T00:54:46.636335+0000 (oldest deadline 2024-03-10T00:55:10.136395+0000) 2024-03-10T00:55:11.180 INFO:journalctl@ceph.osd.7.smithi170.stdout:Mar 10 00:55:11 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-osd-7[47108]: 2024-03-10T00:55:11.053+0000 7f7b81779700 -1 osd.7 71 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.24440.0:149 2.0 2:353ec8f9:::gateway.conf:head [getxattr epoch in=5b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e71) 2024-03-10T00:55:11.524 INFO:journalctl@ceph.osd.6.smithi170.stdout:Mar 10 00:55:11 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-osd-6[43049]: 2024-03-10T00:55:11.268+0000 7f5852423700 -1 osd.6 71 heartbeat_check: no reply from 172.21.15.106:6806 osd.0 since back 2024-03-10T00:54:17.852283+0000 front 2024-03-10T00:54:17.852086+0000 (oldest deadline 2024-03-10T00:54:43.752263+0000) 2024-03-10T00:55:11.524 INFO:journalctl@ceph.osd.6.smithi170.stdout:Mar 10 00:55:11 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-osd-6[43049]: 2024-03-10T00:55:11.268+0000 7f5852423700 -1 osd.6 71 heartbeat_check: no reply from 172.21.15.106:6814 osd.1 since back 2024-03-10T00:54:28.453247+0000 front 2024-03-10T00:54:28.453219+0000 (oldest deadline 2024-03-10T00:54:53.753144+0000) 2024-03-10T00:55:11.524 INFO:journalctl@ceph.osd.6.smithi170.stdout:Mar 10 00:55:11 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-osd-6[43049]: 2024-03-10T00:55:11.268+0000 7f5852423700 -1 osd.6 71 heartbeat_check: no reply from 172.21.15.106:6822 osd.2 since back 2024-03-10T00:54:38.954861+0000 front 2024-03-10T00:54:38.954901+0000 (oldest deadline 2024-03-10T00:55:03.654750+0000) 2024-03-10T00:55:11.524 INFO:journalctl@ceph.osd.6.smithi170.stdout:Mar 10 00:55:11 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-osd-6[43049]: 2024-03-10T00:55:11.268+0000 7f5852423700 -1 osd.6 71 heartbeat_check: no reply from 172.21.15.106:6830 osd.3 since back 2024-03-10T00:54:45.955671+0000 front 2024-03-10T00:54:45.955712+0000 (oldest deadline 2024-03-10T00:55:10.055578+0000) 2024-03-10T00:55:12.274 INFO:journalctl@ceph.osd.7.smithi170.stdout:Mar 10 00:55:12 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-osd-7[47108]: 2024-03-10T00:55:12.020+0000 7f7b81779700 -1 osd.7 71 heartbeat_check: no reply from 172.21.15.106:6806 osd.0 since back 2024-03-10T00:54:19.432267+0000 front 2024-03-10T00:54:19.432106+0000 (oldest deadline 2024-03-10T00:54:43.532104+0000) 2024-03-10T00:55:12.274 INFO:journalctl@ceph.osd.7.smithi170.stdout:Mar 10 00:55:12 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-osd-7[47108]: 2024-03-10T00:55:12.020+0000 7f7b81779700 -1 osd.7 71 heartbeat_check: no reply from 172.21.15.106:6814 osd.1 since back 2024-03-10T00:54:30.533390+0000 front 2024-03-10T00:54:30.533458+0000 (oldest deadline 2024-03-10T00:54:54.633272+0000) 2024-03-10T00:55:12.274 INFO:journalctl@ceph.osd.7.smithi170.stdout:Mar 10 00:55:12 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-osd-7[47108]: 2024-03-10T00:55:12.020+0000 7f7b81779700 -1 osd.7 71 heartbeat_check: no reply from 172.21.15.106:6822 osd.2 since back 2024-03-10T00:54:38.435593+0000 front 2024-03-10T00:54:38.435565+0000 (oldest deadline 2024-03-10T00:55:01.935552+0000) 2024-03-10T00:55:12.274 INFO:journalctl@ceph.osd.7.smithi170.stdout:Mar 10 00:55:12 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-osd-7[47108]: 2024-03-10T00:55:12.020+0000 7f7b81779700 -1 osd.7 71 heartbeat_check: no reply from 172.21.15.106:6830 osd.3 since back 2024-03-10T00:54:46.636263+0000 front 2024-03-10T00:54:46.636335+0000 (oldest deadline 2024-03-10T00:55:10.136395+0000) 2024-03-10T00:55:12.274 INFO:journalctl@ceph.osd.7.smithi170.stdout:Mar 10 00:55:12 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-osd-7[47108]: 2024-03-10T00:55:12.020+0000 7f7b81779700 -1 osd.7 71 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.24440.0:149 2.0 2:353ec8f9:::gateway.conf:head [getxattr epoch in=5b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e71) 2024-03-10T00:55:12.754 INFO:journalctl@ceph.osd.6.smithi170.stdout:Mar 10 00:55:12 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-osd-6[43049]: 2024-03-10T00:55:12.291+0000 7f5852423700 -1 osd.6 71 heartbeat_check: no reply from 172.21.15.106:6806 osd.0 since back 2024-03-10T00:54:17.852283+0000 front 2024-03-10T00:54:17.852086+0000 (oldest deadline 2024-03-10T00:54:43.752263+0000) 2024-03-10T00:55:12.754 INFO:journalctl@ceph.osd.6.smithi170.stdout:Mar 10 00:55:12 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-osd-6[43049]: 2024-03-10T00:55:12.291+0000 7f5852423700 -1 osd.6 71 heartbeat_check: no reply from 172.21.15.106:6814 osd.1 since back 2024-03-10T00:54:28.453247+0000 front 2024-03-10T00:54:28.453219+0000 (oldest deadline 2024-03-10T00:54:53.753144+0000) 2024-03-10T00:55:12.754 INFO:journalctl@ceph.osd.6.smithi170.stdout:Mar 10 00:55:12 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-osd-6[43049]: 2024-03-10T00:55:12.291+0000 7f5852423700 -1 osd.6 71 heartbeat_check: no reply from 172.21.15.106:6822 osd.2 since back 2024-03-10T00:54:38.954861+0000 front 2024-03-10T00:54:38.954901+0000 (oldest deadline 2024-03-10T00:55:03.654750+0000) 2024-03-10T00:55:12.754 INFO:journalctl@ceph.osd.6.smithi170.stdout:Mar 10 00:55:12 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-osd-6[43049]: 2024-03-10T00:55:12.291+0000 7f5852423700 -1 osd.6 71 heartbeat_check: no reply from 172.21.15.106:6830 osd.3 since back 2024-03-10T00:54:45.955671+0000 front 2024-03-10T00:54:45.955712+0000 (oldest deadline 2024-03-10T00:55:10.055578+0000) 2024-03-10T00:55:13.017 INFO:journalctl@ceph.osd.7.smithi170.stdout:Mar 10 00:55:13 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-osd-7[47108]: 2024-03-10T00:55:13.004+0000 7f7b81779700 -1 osd.7 71 heartbeat_check: no reply from 172.21.15.106:6806 osd.0 since back 2024-03-10T00:54:19.432267+0000 front 2024-03-10T00:54:19.432106+0000 (oldest deadline 2024-03-10T00:54:43.532104+0000) 2024-03-10T00:55:13.018 INFO:journalctl@ceph.osd.7.smithi170.stdout:Mar 10 00:55:13 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-osd-7[47108]: 2024-03-10T00:55:13.004+0000 7f7b81779700 -1 osd.7 71 heartbeat_check: no reply from 172.21.15.106:6814 osd.1 since back 2024-03-10T00:54:30.533390+0000 front 2024-03-10T00:54:30.533458+0000 (oldest deadline 2024-03-10T00:54:54.633272+0000) 2024-03-10T00:55:13.018 INFO:journalctl@ceph.osd.7.smithi170.stdout:Mar 10 00:55:13 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-osd-7[47108]: 2024-03-10T00:55:13.004+0000 7f7b81779700 -1 osd.7 71 heartbeat_check: no reply from 172.21.15.106:6822 osd.2 since back 2024-03-10T00:54:38.435593+0000 front 2024-03-10T00:54:38.435565+0000 (oldest deadline 2024-03-10T00:55:01.935552+0000) 2024-03-10T00:55:13.018 INFO:journalctl@ceph.osd.7.smithi170.stdout:Mar 10 00:55:13 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-osd-7[47108]: 2024-03-10T00:55:13.004+0000 7f7b81779700 -1 osd.7 71 heartbeat_check: no reply from 172.21.15.106:6830 osd.3 since back 2024-03-10T00:54:46.636263+0000 front 2024-03-10T00:54:46.636335+0000 (oldest deadline 2024-03-10T00:55:10.136395+0000) 2024-03-10T00:55:13.018 INFO:journalctl@ceph.osd.7.smithi170.stdout:Mar 10 00:55:13 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-osd-7[47108]: 2024-03-10T00:55:13.004+0000 7f7b81779700 -1 osd.7 71 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.24440.0:149 2.0 2:353ec8f9:::gateway.conf:head [getxattr epoch in=5b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e71) 2024-03-10T00:55:13.274 INFO:journalctl@ceph.osd.6.smithi170.stdout:Mar 10 00:55:13 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-osd-6[43049]: 2024-03-10T00:55:13.261+0000 7f5852423700 -1 osd.6 71 heartbeat_check: no reply from 172.21.15.106:6806 osd.0 since back 2024-03-10T00:54:17.852283+0000 front 2024-03-10T00:54:17.852086+0000 (oldest deadline 2024-03-10T00:54:43.752263+0000) 2024-03-10T00:55:13.274 INFO:journalctl@ceph.osd.6.smithi170.stdout:Mar 10 00:55:13 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-osd-6[43049]: 2024-03-10T00:55:13.261+0000 7f5852423700 -1 osd.6 71 heartbeat_check: no reply from 172.21.15.106:6814 osd.1 since back 2024-03-10T00:54:28.453247+0000 front 2024-03-10T00:54:28.453219+0000 (oldest deadline 2024-03-10T00:54:53.753144+0000) 2024-03-10T00:55:13.274 INFO:journalctl@ceph.osd.6.smithi170.stdout:Mar 10 00:55:13 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-osd-6[43049]: 2024-03-10T00:55:13.261+0000 7f5852423700 -1 osd.6 71 heartbeat_check: no reply from 172.21.15.106:6822 osd.2 since back 2024-03-10T00:54:38.954861+0000 front 2024-03-10T00:54:38.954901+0000 (oldest deadline 2024-03-10T00:55:03.654750+0000) 2024-03-10T00:55:13.274 INFO:journalctl@ceph.osd.6.smithi170.stdout:Mar 10 00:55:13 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-osd-6[43049]: 2024-03-10T00:55:13.261+0000 7f5852423700 -1 osd.6 71 heartbeat_check: no reply from 172.21.15.106:6830 osd.3 since back 2024-03-10T00:54:45.955671+0000 front 2024-03-10T00:54:45.955712+0000 (oldest deadline 2024-03-10T00:55:10.055578+0000) 2024-03-10T00:55:14.236 INFO:journalctl@ceph.osd.7.smithi170.stdout:Mar 10 00:55:13 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-osd-7[47108]: 2024-03-10T00:55:13.964+0000 7f7b81779700 -1 osd.7 71 heartbeat_check: no reply from 172.21.15.106:6806 osd.0 since back 2024-03-10T00:54:19.432267+0000 front 2024-03-10T00:54:19.432106+0000 (oldest deadline 2024-03-10T00:54:43.532104+0000) 2024-03-10T00:55:14.237 INFO:journalctl@ceph.osd.7.smithi170.stdout:Mar 10 00:55:13 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-osd-7[47108]: 2024-03-10T00:55:13.964+0000 7f7b81779700 -1 osd.7 71 heartbeat_check: no reply from 172.21.15.106:6814 osd.1 since back 2024-03-10T00:54:30.533390+0000 front 2024-03-10T00:54:30.533458+0000 (oldest deadline 2024-03-10T00:54:54.633272+0000) 2024-03-10T00:55:14.237 INFO:journalctl@ceph.osd.7.smithi170.stdout:Mar 10 00:55:13 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-osd-7[47108]: 2024-03-10T00:55:13.964+0000 7f7b81779700 -1 osd.7 71 heartbeat_check: no reply from 172.21.15.106:6822 osd.2 since back 2024-03-10T00:54:38.435593+0000 front 2024-03-10T00:54:38.435565+0000 (oldest deadline 2024-03-10T00:55:01.935552+0000) 2024-03-10T00:55:14.237 INFO:journalctl@ceph.osd.7.smithi170.stdout:Mar 10 00:55:13 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-osd-7[47108]: 2024-03-10T00:55:13.964+0000 7f7b81779700 -1 osd.7 71 heartbeat_check: no reply from 172.21.15.106:6830 osd.3 since back 2024-03-10T00:54:46.636263+0000 front 2024-03-10T00:54:46.636335+0000 (oldest deadline 2024-03-10T00:55:10.136395+0000) 2024-03-10T00:55:14.237 INFO:journalctl@ceph.osd.7.smithi170.stdout:Mar 10 00:55:13 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-osd-7[47108]: 2024-03-10T00:55:13.964+0000 7f7b81779700 -1 osd.7 71 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.24440.0:149 2.0 2:353ec8f9:::gateway.conf:head [getxattr epoch in=5b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e71) 2024-03-10T00:55:14.985 INFO:journalctl@ceph.osd.6.smithi170.stdout:Mar 10 00:55:14 smithi170 podman[66996]: 2024-03-10 00:55:14.731246877 +0000 UTC m=+5.882092197 container died b3230350a17e21dd1107c2f6c4df759bd53be495a28f151669d1a8fd3ca5405c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bc4a24690ce5d7255d490e21058d8c114b55e1b93a3d8345f90b6700ad29dada, name=ceph-75c37498-de77-11ee-95c7-87774f69a715-osd-6, GIT_BRANCH=HEAD, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240222, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, RELEASE=wip-yuri4-testing-2024-03-05-0854-831bc18) 2024-03-10T00:55:15.274 INFO:journalctl@ceph.osd.6.smithi170.stdout:Mar 10 00:55:15 smithi170 podman[66996]: 2024-03-10 00:55:15.092260498 +0000 UTC m=+6.243105818 container cleanup b3230350a17e21dd1107c2f6c4df759bd53be495a28f151669d1a8fd3ca5405c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bc4a24690ce5d7255d490e21058d8c114b55e1b93a3d8345f90b6700ad29dada, name=ceph-75c37498-de77-11ee-95c7-87774f69a715-osd-6, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, RELEASE=wip-yuri4-testing-2024-03-05-0854-831bc18, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, ceph=True, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.build-date=20240222) 2024-03-10T00:55:15.274 INFO:journalctl@ceph.osd.6.smithi170.stdout:Mar 10 00:55:15 smithi170 bash[66996]: ceph-75c37498-de77-11ee-95c7-87774f69a715-osd-6 2024-03-10T00:55:15.274 INFO:journalctl@ceph.osd.7.smithi170.stdout:Mar 10 00:55:14 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-osd-7[47108]: 2024-03-10T00:55:14.987+0000 7f7b81779700 -1 osd.7 71 heartbeat_check: no reply from 172.21.15.106:6806 osd.0 since back 2024-03-10T00:54:19.432267+0000 front 2024-03-10T00:54:19.432106+0000 (oldest deadline 2024-03-10T00:54:43.532104+0000) 2024-03-10T00:55:15.274 INFO:journalctl@ceph.osd.7.smithi170.stdout:Mar 10 00:55:14 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-osd-7[47108]: 2024-03-10T00:55:14.987+0000 7f7b81779700 -1 osd.7 71 heartbeat_check: no reply from 172.21.15.106:6814 osd.1 since back 2024-03-10T00:54:30.533390+0000 front 2024-03-10T00:54:30.533458+0000 (oldest deadline 2024-03-10T00:54:54.633272+0000) 2024-03-10T00:55:15.274 INFO:journalctl@ceph.osd.7.smithi170.stdout:Mar 10 00:55:14 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-osd-7[47108]: 2024-03-10T00:55:14.987+0000 7f7b81779700 -1 osd.7 71 heartbeat_check: no reply from 172.21.15.106:6822 osd.2 since back 2024-03-10T00:54:38.435593+0000 front 2024-03-10T00:54:38.435565+0000 (oldest deadline 2024-03-10T00:55:01.935552+0000) 2024-03-10T00:55:15.274 INFO:journalctl@ceph.osd.7.smithi170.stdout:Mar 10 00:55:14 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-osd-7[47108]: 2024-03-10T00:55:14.987+0000 7f7b81779700 -1 osd.7 71 heartbeat_check: no reply from 172.21.15.106:6830 osd.3 since back 2024-03-10T00:54:46.636263+0000 front 2024-03-10T00:54:46.636335+0000 (oldest deadline 2024-03-10T00:55:10.136395+0000) 2024-03-10T00:55:15.274 INFO:journalctl@ceph.osd.7.smithi170.stdout:Mar 10 00:55:14 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-osd-7[47108]: 2024-03-10T00:55:14.987+0000 7f7b81779700 -1 osd.7 71 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.24440.0:149 2.0 2:353ec8f9:::gateway.conf:head [getxattr epoch in=5b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e71) 2024-03-10T00:55:15.652 INFO:journalctl@ceph.osd.6.smithi170.stdout:Mar 10 00:55:15 smithi170 podman[67555]: 2024-03-10 00:55:15.34335693 +0000 UTC m=+1.041802388 container remove b3230350a17e21dd1107c2f6c4df759bd53be495a28f151669d1a8fd3ca5405c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bc4a24690ce5d7255d490e21058d8c114b55e1b93a3d8345f90b6700ad29dada, name=ceph-75c37498-de77-11ee-95c7-87774f69a715-osd-6, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-yuri4-testing-2024-03-05-0854-831bc18, org.label-schema.build-date=20240222, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2) 2024-03-10T00:55:15.968 INFO:journalctl@ceph.osd.6.smithi170.stdout:Mar 10 00:55:15 smithi170 podman[67652]: 2024-03-10 00:55:15.558541086 +0000 UTC m=+0.017663105 image pull 01a4f6a6bb3e92ab9ab39a2f835684ea9b10e6ec9fb81ac08ebea4165583a538 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bc4a24690ce5d7255d490e21058d8c114b55e1b93a3d8345f90b6700ad29dada 2024-03-10T00:55:15.968 INFO:journalctl@ceph.osd.6.smithi170.stdout:Mar 10 00:55:15 smithi170 podman[67652]: 2024-03-10 00:55:15.66925087 +0000 UTC m=+0.128372927 container create c955b8609ab30eaa75ec46bbab134a7b59c9066dc34df1ee447135eb52f93362 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bc4a24690ce5d7255d490e21058d8c114b55e1b93a3d8345f90b6700ad29dada, name=ceph-75c37498-de77-11ee-95c7-87774f69a715-osd-6-deactivate, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, RELEASE=wip-yuri4-testing-2024-03-05-0854-831bc18, ceph=True, org.label-schema.license=GPLv2, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, org.label-schema.build-date=20240222, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS) 2024-03-10T00:55:15.968 INFO:journalctl@ceph.osd.6.smithi170.stdout:Mar 10 00:55:15 smithi170 podman[67652]: 2024-03-10 00:55:15.871672974 +0000 UTC m=+0.330794981 container init c955b8609ab30eaa75ec46bbab134a7b59c9066dc34df1ee447135eb52f93362 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bc4a24690ce5d7255d490e21058d8c114b55e1b93a3d8345f90b6700ad29dada, name=ceph-75c37498-de77-11ee-95c7-87774f69a715-osd-6-deactivate, org.label-schema.build-date=20240222, RELEASE=wip-yuri4-testing-2024-03-05-0854-831bc18, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2) 2024-03-10T00:55:15.969 INFO:journalctl@ceph.osd.6.smithi170.stdout:Mar 10 00:55:15 smithi170 podman[67652]: 2024-03-10 00:55:15.879840002 +0000 UTC m=+0.338962026 container start c955b8609ab30eaa75ec46bbab134a7b59c9066dc34df1ee447135eb52f93362 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bc4a24690ce5d7255d490e21058d8c114b55e1b93a3d8345f90b6700ad29dada, name=ceph-75c37498-de77-11ee-95c7-87774f69a715-osd-6-deactivate, io.buildah.version=1.33.5, ceph=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, RELEASE=wip-yuri4-testing-2024-03-05-0854-831bc18, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240222, CEPH_POINT_RELEASE=) 2024-03-10T00:55:15.969 INFO:journalctl@ceph.osd.6.smithi170.stdout:Mar 10 00:55:15 smithi170 podman[67652]: 2024-03-10 00:55:15.971459895 +0000 UTC m=+0.430581924 container attach c955b8609ab30eaa75ec46bbab134a7b59c9066dc34df1ee447135eb52f93362 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bc4a24690ce5d7255d490e21058d8c114b55e1b93a3d8345f90b6700ad29dada, name=ceph-75c37498-de77-11ee-95c7-87774f69a715-osd-6-deactivate, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.build-date=20240222, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, RELEASE=wip-yuri4-testing-2024-03-05-0854-831bc18, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, ceph=True) 2024-03-10T00:55:16.274 INFO:journalctl@ceph.osd.6.smithi170.stdout:Mar 10 00:55:16 smithi170 podman[67652]: 2024-03-10 00:55:16.190109707 +0000 UTC m=+0.649231721 container died c955b8609ab30eaa75ec46bbab134a7b59c9066dc34df1ee447135eb52f93362 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bc4a24690ce5d7255d490e21058d8c114b55e1b93a3d8345f90b6700ad29dada, name=ceph-75c37498-de77-11ee-95c7-87774f69a715-osd-6-deactivate, io.buildah.version=1.33.5, org.label-schema.build-date=20240222, ceph=True, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, RELEASE=wip-yuri4-testing-2024-03-05-0854-831bc18) 2024-03-10T00:55:16.274 INFO:journalctl@ceph.osd.7.smithi170.stdout:Mar 10 00:55:16 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-osd-7[47108]: 2024-03-10T00:55:16.037+0000 7f7b81779700 -1 osd.7 71 heartbeat_check: no reply from 172.21.15.106:6806 osd.0 since back 2024-03-10T00:54:19.432267+0000 front 2024-03-10T00:54:19.432106+0000 (oldest deadline 2024-03-10T00:54:43.532104+0000) 2024-03-10T00:55:16.274 INFO:journalctl@ceph.osd.7.smithi170.stdout:Mar 10 00:55:16 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-osd-7[47108]: 2024-03-10T00:55:16.037+0000 7f7b81779700 -1 osd.7 71 heartbeat_check: no reply from 172.21.15.106:6814 osd.1 since back 2024-03-10T00:54:30.533390+0000 front 2024-03-10T00:54:30.533458+0000 (oldest deadline 2024-03-10T00:54:54.633272+0000) 2024-03-10T00:55:16.274 INFO:journalctl@ceph.osd.7.smithi170.stdout:Mar 10 00:55:16 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-osd-7[47108]: 2024-03-10T00:55:16.037+0000 7f7b81779700 -1 osd.7 71 heartbeat_check: no reply from 172.21.15.106:6822 osd.2 since back 2024-03-10T00:54:38.435593+0000 front 2024-03-10T00:54:38.435565+0000 (oldest deadline 2024-03-10T00:55:01.935552+0000) 2024-03-10T00:55:16.274 INFO:journalctl@ceph.osd.7.smithi170.stdout:Mar 10 00:55:16 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-osd-7[47108]: 2024-03-10T00:55:16.037+0000 7f7b81779700 -1 osd.7 71 heartbeat_check: no reply from 172.21.15.106:6830 osd.3 since back 2024-03-10T00:54:46.636263+0000 front 2024-03-10T00:54:46.636335+0000 (oldest deadline 2024-03-10T00:55:10.136395+0000) 2024-03-10T00:55:16.274 INFO:journalctl@ceph.osd.7.smithi170.stdout:Mar 10 00:55:16 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-osd-7[47108]: 2024-03-10T00:55:16.037+0000 7f7b81779700 -1 osd.7 71 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.24440.0:149 2.0 2:353ec8f9:::gateway.conf:head [getxattr epoch in=5b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e71) 2024-03-10T00:55:16.874 DEBUG:teuthology.orchestra.run.smithi170:> sudo pkill -f 'journalctl -f -n 0 -u ceph-75c37498-de77-11ee-95c7-87774f69a715@osd.6.service' 2024-03-10T00:55:16.917 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-10T00:55:16.917 INFO:tasks.cephadm.osd.6:Stopped osd.6 2024-03-10T00:55:16.918 INFO:tasks.cephadm.osd.7:Stopping osd.7... 2024-03-10T00:55:16.918 DEBUG:teuthology.orchestra.run.smithi170:> sudo systemctl stop ceph-75c37498-de77-11ee-95c7-87774f69a715@osd.7 2024-03-10T00:55:17.245 INFO:journalctl@ceph.osd.7.smithi170.stdout:Mar 10 00:55:16 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-osd-7[47108]: 2024-03-10T00:55:16.988+0000 7f7b81779700 -1 osd.7 71 heartbeat_check: no reply from 172.21.15.106:6806 osd.0 since back 2024-03-10T00:54:19.432267+0000 front 2024-03-10T00:54:19.432106+0000 (oldest deadline 2024-03-10T00:54:43.532104+0000) 2024-03-10T00:55:17.245 INFO:journalctl@ceph.osd.7.smithi170.stdout:Mar 10 00:55:16 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-osd-7[47108]: 2024-03-10T00:55:16.988+0000 7f7b81779700 -1 osd.7 71 heartbeat_check: no reply from 172.21.15.106:6814 osd.1 since back 2024-03-10T00:54:30.533390+0000 front 2024-03-10T00:54:30.533458+0000 (oldest deadline 2024-03-10T00:54:54.633272+0000) 2024-03-10T00:55:17.245 INFO:journalctl@ceph.osd.7.smithi170.stdout:Mar 10 00:55:16 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-osd-7[47108]: 2024-03-10T00:55:16.988+0000 7f7b81779700 -1 osd.7 71 heartbeat_check: no reply from 172.21.15.106:6822 osd.2 since back 2024-03-10T00:54:38.435593+0000 front 2024-03-10T00:54:38.435565+0000 (oldest deadline 2024-03-10T00:55:01.935552+0000) 2024-03-10T00:55:17.245 INFO:journalctl@ceph.osd.7.smithi170.stdout:Mar 10 00:55:16 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-osd-7[47108]: 2024-03-10T00:55:16.988+0000 7f7b81779700 -1 osd.7 71 heartbeat_check: no reply from 172.21.15.106:6830 osd.3 since back 2024-03-10T00:54:46.636263+0000 front 2024-03-10T00:54:46.636335+0000 (oldest deadline 2024-03-10T00:55:10.136395+0000) 2024-03-10T00:55:17.245 INFO:journalctl@ceph.osd.7.smithi170.stdout:Mar 10 00:55:16 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-osd-7[47108]: 2024-03-10T00:55:16.988+0000 7f7b81779700 -1 osd.7 71 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.24440.0:149 2.0 2:353ec8f9:::gateway.conf:head [getxattr epoch in=5b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e71) 2024-03-10T00:55:17.246 INFO:journalctl@ceph.osd.7.smithi170.stdout:Mar 10 00:55:16 smithi170 systemd[1]: Stopping Ceph osd.7 for 75c37498-de77-11ee-95c7-87774f69a715... 2024-03-10T00:55:17.524 INFO:journalctl@ceph.osd.7.smithi170.stdout:Mar 10 00:55:17 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-osd-7[47108]: 2024-03-10T00:55:17.247+0000 7f7b8918d700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.7 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-10T00:55:17.524 INFO:journalctl@ceph.osd.7.smithi170.stdout:Mar 10 00:55:17 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-osd-7[47108]: 2024-03-10T00:55:17.247+0000 7f7b8918d700 -1 osd.7 71 *** Got signal Terminated *** 2024-03-10T00:55:17.524 INFO:journalctl@ceph.osd.7.smithi170.stdout:Mar 10 00:55:17 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-osd-7[47108]: 2024-03-10T00:55:17.247+0000 7f7b8918d700 -1 osd.7 71 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-10T00:55:18.274 INFO:journalctl@ceph.osd.7.smithi170.stdout:Mar 10 00:55:17 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-osd-7[47108]: 2024-03-10T00:55:17.961+0000 7f7b81779700 -1 osd.7 71 heartbeat_check: no reply from 172.21.15.106:6806 osd.0 since back 2024-03-10T00:54:19.432267+0000 front 2024-03-10T00:54:19.432106+0000 (oldest deadline 2024-03-10T00:54:43.532104+0000) 2024-03-10T00:55:18.274 INFO:journalctl@ceph.osd.7.smithi170.stdout:Mar 10 00:55:17 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-osd-7[47108]: 2024-03-10T00:55:17.961+0000 7f7b81779700 -1 osd.7 71 heartbeat_check: no reply from 172.21.15.106:6814 osd.1 since back 2024-03-10T00:54:30.533390+0000 front 2024-03-10T00:54:30.533458+0000 (oldest deadline 2024-03-10T00:54:54.633272+0000) 2024-03-10T00:55:18.274 INFO:journalctl@ceph.osd.7.smithi170.stdout:Mar 10 00:55:17 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-osd-7[47108]: 2024-03-10T00:55:17.961+0000 7f7b81779700 -1 osd.7 71 heartbeat_check: no reply from 172.21.15.106:6822 osd.2 since back 2024-03-10T00:54:38.435593+0000 front 2024-03-10T00:54:38.435565+0000 (oldest deadline 2024-03-10T00:55:01.935552+0000) 2024-03-10T00:55:18.274 INFO:journalctl@ceph.osd.7.smithi170.stdout:Mar 10 00:55:17 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-osd-7[47108]: 2024-03-10T00:55:17.961+0000 7f7b81779700 -1 osd.7 71 heartbeat_check: no reply from 172.21.15.106:6830 osd.3 since back 2024-03-10T00:54:46.636263+0000 front 2024-03-10T00:54:46.636335+0000 (oldest deadline 2024-03-10T00:55:10.136395+0000) 2024-03-10T00:55:18.274 INFO:journalctl@ceph.osd.7.smithi170.stdout:Mar 10 00:55:17 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-osd-7[47108]: 2024-03-10T00:55:17.961+0000 7f7b81779700 -1 osd.7 71 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.24440.0:149 2.0 2:353ec8f9:::gateway.conf:head [getxattr epoch in=5b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e71) 2024-03-10T00:55:19.274 INFO:journalctl@ceph.osd.7.smithi170.stdout:Mar 10 00:55:18 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-osd-7[47108]: 2024-03-10T00:55:18.912+0000 7f7b81779700 -1 osd.7 71 heartbeat_check: no reply from 172.21.15.106:6806 osd.0 since back 2024-03-10T00:54:19.432267+0000 front 2024-03-10T00:54:19.432106+0000 (oldest deadline 2024-03-10T00:54:43.532104+0000) 2024-03-10T00:55:19.274 INFO:journalctl@ceph.osd.7.smithi170.stdout:Mar 10 00:55:18 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-osd-7[47108]: 2024-03-10T00:55:18.912+0000 7f7b81779700 -1 osd.7 71 heartbeat_check: no reply from 172.21.15.106:6814 osd.1 since back 2024-03-10T00:54:30.533390+0000 front 2024-03-10T00:54:30.533458+0000 (oldest deadline 2024-03-10T00:54:54.633272+0000) 2024-03-10T00:55:19.274 INFO:journalctl@ceph.osd.7.smithi170.stdout:Mar 10 00:55:18 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-osd-7[47108]: 2024-03-10T00:55:18.912+0000 7f7b81779700 -1 osd.7 71 heartbeat_check: no reply from 172.21.15.106:6822 osd.2 since back 2024-03-10T00:54:38.435593+0000 front 2024-03-10T00:54:38.435565+0000 (oldest deadline 2024-03-10T00:55:01.935552+0000) 2024-03-10T00:55:19.274 INFO:journalctl@ceph.osd.7.smithi170.stdout:Mar 10 00:55:18 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-osd-7[47108]: 2024-03-10T00:55:18.912+0000 7f7b81779700 -1 osd.7 71 heartbeat_check: no reply from 172.21.15.106:6830 osd.3 since back 2024-03-10T00:54:46.636263+0000 front 2024-03-10T00:54:46.636335+0000 (oldest deadline 2024-03-10T00:55:10.136395+0000) 2024-03-10T00:55:19.274 INFO:journalctl@ceph.osd.7.smithi170.stdout:Mar 10 00:55:18 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-osd-7[47108]: 2024-03-10T00:55:18.912+0000 7f7b81779700 -1 osd.7 71 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.24440.0:149 2.0 2:353ec8f9:::gateway.conf:head [getxattr epoch in=5b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e71) 2024-03-10T00:55:20.274 INFO:journalctl@ceph.osd.7.smithi170.stdout:Mar 10 00:55:19 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-osd-7[47108]: 2024-03-10T00:55:19.941+0000 7f7b81779700 -1 osd.7 71 heartbeat_check: no reply from 172.21.15.106:6806 osd.0 since back 2024-03-10T00:54:19.432267+0000 front 2024-03-10T00:54:19.432106+0000 (oldest deadline 2024-03-10T00:54:43.532104+0000) 2024-03-10T00:55:20.274 INFO:journalctl@ceph.osd.7.smithi170.stdout:Mar 10 00:55:19 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-osd-7[47108]: 2024-03-10T00:55:19.941+0000 7f7b81779700 -1 osd.7 71 heartbeat_check: no reply from 172.21.15.106:6814 osd.1 since back 2024-03-10T00:54:30.533390+0000 front 2024-03-10T00:54:30.533458+0000 (oldest deadline 2024-03-10T00:54:54.633272+0000) 2024-03-10T00:55:20.274 INFO:journalctl@ceph.osd.7.smithi170.stdout:Mar 10 00:55:19 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-osd-7[47108]: 2024-03-10T00:55:19.941+0000 7f7b81779700 -1 osd.7 71 heartbeat_check: no reply from 172.21.15.106:6822 osd.2 since back 2024-03-10T00:54:38.435593+0000 front 2024-03-10T00:54:38.435565+0000 (oldest deadline 2024-03-10T00:55:01.935552+0000) 2024-03-10T00:55:20.274 INFO:journalctl@ceph.osd.7.smithi170.stdout:Mar 10 00:55:19 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-osd-7[47108]: 2024-03-10T00:55:19.941+0000 7f7b81779700 -1 osd.7 71 heartbeat_check: no reply from 172.21.15.106:6830 osd.3 since back 2024-03-10T00:54:46.636263+0000 front 2024-03-10T00:54:46.636335+0000 (oldest deadline 2024-03-10T00:55:10.136395+0000) 2024-03-10T00:55:20.274 INFO:journalctl@ceph.osd.7.smithi170.stdout:Mar 10 00:55:19 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-osd-7[47108]: 2024-03-10T00:55:19.941+0000 7f7b81779700 -1 osd.7 71 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.24440.0:149 2.0 2:353ec8f9:::gateway.conf:head [getxattr epoch in=5b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e71) 2024-03-10T00:55:21.274 INFO:journalctl@ceph.osd.7.smithi170.stdout:Mar 10 00:55:20 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-osd-7[47108]: 2024-03-10T00:55:20.923+0000 7f7b81779700 -1 osd.7 71 heartbeat_check: no reply from 172.21.15.106:6806 osd.0 since back 2024-03-10T00:54:19.432267+0000 front 2024-03-10T00:54:19.432106+0000 (oldest deadline 2024-03-10T00:54:43.532104+0000) 2024-03-10T00:55:21.274 INFO:journalctl@ceph.osd.7.smithi170.stdout:Mar 10 00:55:20 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-osd-7[47108]: 2024-03-10T00:55:20.923+0000 7f7b81779700 -1 osd.7 71 heartbeat_check: no reply from 172.21.15.106:6814 osd.1 since back 2024-03-10T00:54:30.533390+0000 front 2024-03-10T00:54:30.533458+0000 (oldest deadline 2024-03-10T00:54:54.633272+0000) 2024-03-10T00:55:21.274 INFO:journalctl@ceph.osd.7.smithi170.stdout:Mar 10 00:55:20 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-osd-7[47108]: 2024-03-10T00:55:20.923+0000 7f7b81779700 -1 osd.7 71 heartbeat_check: no reply from 172.21.15.106:6822 osd.2 since back 2024-03-10T00:54:38.435593+0000 front 2024-03-10T00:54:38.435565+0000 (oldest deadline 2024-03-10T00:55:01.935552+0000) 2024-03-10T00:55:21.274 INFO:journalctl@ceph.osd.7.smithi170.stdout:Mar 10 00:55:20 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-osd-7[47108]: 2024-03-10T00:55:20.923+0000 7f7b81779700 -1 osd.7 71 heartbeat_check: no reply from 172.21.15.106:6830 osd.3 since back 2024-03-10T00:54:46.636263+0000 front 2024-03-10T00:54:46.636335+0000 (oldest deadline 2024-03-10T00:55:10.136395+0000) 2024-03-10T00:55:21.275 INFO:journalctl@ceph.osd.7.smithi170.stdout:Mar 10 00:55:20 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-osd-7[47108]: 2024-03-10T00:55:20.923+0000 7f7b81779700 -1 osd.7 71 heartbeat_check: no reply from 172.21.15.170:6804 osd.4 since back 2024-03-10T00:54:55.937471+0000 front 2024-03-10T00:54:55.937503+0000 (oldest deadline 2024-03-10T00:55:20.637564+0000) 2024-03-10T00:55:21.275 INFO:journalctl@ceph.osd.7.smithi170.stdout:Mar 10 00:55:20 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-osd-7[47108]: 2024-03-10T00:55:20.923+0000 7f7b81779700 -1 osd.7 71 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.24440.0:149 2.0 2:353ec8f9:::gateway.conf:head [getxattr epoch in=5b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e71) 2024-03-10T00:55:22.274 INFO:journalctl@ceph.osd.7.smithi170.stdout:Mar 10 00:55:21 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-osd-7[47108]: 2024-03-10T00:55:21.970+0000 7f7b81779700 -1 osd.7 71 heartbeat_check: no reply from 172.21.15.106:6806 osd.0 since back 2024-03-10T00:54:19.432267+0000 front 2024-03-10T00:54:19.432106+0000 (oldest deadline 2024-03-10T00:54:43.532104+0000) 2024-03-10T00:55:22.274 INFO:journalctl@ceph.osd.7.smithi170.stdout:Mar 10 00:55:21 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-osd-7[47108]: 2024-03-10T00:55:21.970+0000 7f7b81779700 -1 osd.7 71 heartbeat_check: no reply from 172.21.15.106:6814 osd.1 since back 2024-03-10T00:54:30.533390+0000 front 2024-03-10T00:54:30.533458+0000 (oldest deadline 2024-03-10T00:54:54.633272+0000) 2024-03-10T00:55:22.274 INFO:journalctl@ceph.osd.7.smithi170.stdout:Mar 10 00:55:21 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-osd-7[47108]: 2024-03-10T00:55:21.970+0000 7f7b81779700 -1 osd.7 71 heartbeat_check: no reply from 172.21.15.106:6822 osd.2 since back 2024-03-10T00:54:38.435593+0000 front 2024-03-10T00:54:38.435565+0000 (oldest deadline 2024-03-10T00:55:01.935552+0000) 2024-03-10T00:55:22.274 INFO:journalctl@ceph.osd.7.smithi170.stdout:Mar 10 00:55:21 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-osd-7[47108]: 2024-03-10T00:55:21.970+0000 7f7b81779700 -1 osd.7 71 heartbeat_check: no reply from 172.21.15.106:6830 osd.3 since back 2024-03-10T00:54:46.636263+0000 front 2024-03-10T00:54:46.636335+0000 (oldest deadline 2024-03-10T00:55:10.136395+0000) 2024-03-10T00:55:22.274 INFO:journalctl@ceph.osd.7.smithi170.stdout:Mar 10 00:55:21 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-osd-7[47108]: 2024-03-10T00:55:21.970+0000 7f7b81779700 -1 osd.7 71 heartbeat_check: no reply from 172.21.15.170:6804 osd.4 since back 2024-03-10T00:54:55.937471+0000 front 2024-03-10T00:54:55.937503+0000 (oldest deadline 2024-03-10T00:55:20.637564+0000) 2024-03-10T00:55:22.274 INFO:journalctl@ceph.osd.7.smithi170.stdout:Mar 10 00:55:21 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-osd-7[47108]: 2024-03-10T00:55:21.970+0000 7f7b81779700 -1 osd.7 71 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.24440.0:149 2.0 2:353ec8f9:::gateway.conf:head [getxattr epoch in=5b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e71) 2024-03-10T00:55:23.000 INFO:journalctl@ceph.osd.7.smithi170.stdout:Mar 10 00:55:22 smithi170 podman[67793]: 2024-03-10 00:55:22.74332443 +0000 UTC m=+5.609615422 container died 4a8bf4bdf3aa235b3078f24d81444c85fa4835003c2804c1e9c79f80e151f9ad (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bc4a24690ce5d7255d490e21058d8c114b55e1b93a3d8345f90b6700ad29dada, name=ceph-75c37498-de77-11ee-95c7-87774f69a715-osd-7, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_CLEAN=True, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, RELEASE=wip-yuri4-testing-2024-03-05-0854-831bc18, org.label-schema.build-date=20240222) 2024-03-10T00:55:23.274 INFO:journalctl@ceph.osd.7.smithi170.stdout:Mar 10 00:55:23 smithi170 podman[67793]: 2024-03-10 00:55:23.169581977 +0000 UTC m=+6.035873000 container cleanup 4a8bf4bdf3aa235b3078f24d81444c85fa4835003c2804c1e9c79f80e151f9ad (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bc4a24690ce5d7255d490e21058d8c114b55e1b93a3d8345f90b6700ad29dada, name=ceph-75c37498-de77-11ee-95c7-87774f69a715-osd-7, org.label-schema.build-date=20240222, org.label-schema.license=GPLv2, ceph=True, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, RELEASE=wip-yuri4-testing-2024-03-05-0854-831bc18) 2024-03-10T00:55:23.274 INFO:journalctl@ceph.osd.7.smithi170.stdout:Mar 10 00:55:23 smithi170 bash[67793]: ceph-75c37498-de77-11ee-95c7-87774f69a715-osd-7 2024-03-10T00:55:23.773 INFO:journalctl@ceph.osd.7.smithi170.stdout:Mar 10 00:55:23 smithi170 podman[67814]: 2024-03-10 00:55:23.387360701 +0000 UTC m=+0.640261724 container remove 4a8bf4bdf3aa235b3078f24d81444c85fa4835003c2804c1e9c79f80e151f9ad (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bc4a24690ce5d7255d490e21058d8c114b55e1b93a3d8345f90b6700ad29dada, name=ceph-75c37498-de77-11ee-95c7-87774f69a715-osd-7, GIT_BRANCH=HEAD, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.build-date=20240222, RELEASE=wip-yuri4-testing-2024-03-05-0854-831bc18, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_CLEAN=True, ceph=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0) 2024-03-10T00:55:24.103 INFO:journalctl@ceph.osd.7.smithi170.stdout:Mar 10 00:55:23 smithi170 podman[67909]: 2024-03-10 00:55:23.729741042 +0000 UTC m=+0.017984340 image pull 01a4f6a6bb3e92ab9ab39a2f835684ea9b10e6ec9fb81ac08ebea4165583a538 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bc4a24690ce5d7255d490e21058d8c114b55e1b93a3d8345f90b6700ad29dada 2024-03-10T00:55:24.103 INFO:journalctl@ceph.osd.7.smithi170.stdout:Mar 10 00:55:23 smithi170 podman[67909]: 2024-03-10 00:55:23.829590438 +0000 UTC m=+0.117833764 container create ca2f2110d766e042f279296ab2c09d176d8a4c94a5bdd06ee097299a55c00c0f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bc4a24690ce5d7255d490e21058d8c114b55e1b93a3d8345f90b6700ad29dada, name=ceph-75c37498-de77-11ee-95c7-87774f69a715-osd-7-deactivate, org.label-schema.license=GPLv2, org.label-schema.build-date=20240222, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, ceph=True, org.label-schema.vendor=CentOS, RELEASE=wip-yuri4-testing-2024-03-05-0854-831bc18, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-10T00:55:24.104 INFO:journalctl@ceph.osd.7.smithi170.stdout:Mar 10 00:55:24 smithi170 podman[67909]: 2024-03-10 00:55:24.015238644 +0000 UTC m=+0.303481923 container init ca2f2110d766e042f279296ab2c09d176d8a4c94a5bdd06ee097299a55c00c0f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bc4a24690ce5d7255d490e21058d8c114b55e1b93a3d8345f90b6700ad29dada, name=ceph-75c37498-de77-11ee-95c7-87774f69a715-osd-7-deactivate, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, org.label-schema.build-date=20240222, GIT_BRANCH=HEAD, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, ceph=True, RELEASE=wip-yuri4-testing-2024-03-05-0854-831bc18) 2024-03-10T00:55:24.104 INFO:journalctl@ceph.osd.7.smithi170.stdout:Mar 10 00:55:24 smithi170 podman[67909]: 2024-03-10 00:55:24.021139286 +0000 UTC m=+0.309382568 container start ca2f2110d766e042f279296ab2c09d176d8a4c94a5bdd06ee097299a55c00c0f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bc4a24690ce5d7255d490e21058d8c114b55e1b93a3d8345f90b6700ad29dada, name=ceph-75c37498-de77-11ee-95c7-87774f69a715-osd-7-deactivate, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-yuri4-testing-2024-03-05-0854-831bc18, ceph=True, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.build-date=20240222, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, CEPH_POINT_RELEASE=) 2024-03-10T00:55:24.104 INFO:journalctl@ceph.osd.7.smithi170.stdout:Mar 10 00:55:24 smithi170 podman[67909]: 2024-03-10 00:55:24.106609137 +0000 UTC m=+0.394852418 container attach ca2f2110d766e042f279296ab2c09d176d8a4c94a5bdd06ee097299a55c00c0f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bc4a24690ce5d7255d490e21058d8c114b55e1b93a3d8345f90b6700ad29dada, name=ceph-75c37498-de77-11ee-95c7-87774f69a715-osd-7-deactivate, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, RELEASE=wip-yuri4-testing-2024-03-05-0854-831bc18, io.buildah.version=1.33.5, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240222, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-10T00:55:24.524 INFO:journalctl@ceph.osd.7.smithi170.stdout:Mar 10 00:55:24 smithi170 podman[67909]: 2024-03-10 00:55:24.310347526 +0000 UTC m=+0.598590816 container died ca2f2110d766e042f279296ab2c09d176d8a4c94a5bdd06ee097299a55c00c0f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bc4a24690ce5d7255d490e21058d8c114b55e1b93a3d8345f90b6700ad29dada, name=ceph-75c37498-de77-11ee-95c7-87774f69a715-osd-7-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240222, CEPH_POINT_RELEASE=, GIT_CLEAN=True, RELEASE=wip-yuri4-testing-2024-03-05-0854-831bc18) 2024-03-10T00:55:24.904 DEBUG:teuthology.orchestra.run.smithi170:> sudo pkill -f 'journalctl -f -n 0 -u ceph-75c37498-de77-11ee-95c7-87774f69a715@osd.7.service' 2024-03-10T00:55:24.946 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-10T00:55:24.946 INFO:tasks.cephadm.osd.7:Stopped osd.7 2024-03-10T00:55:24.946 INFO:tasks.cephadm.ceph.rgw.foo.a:Stopping rgw.foo.a... 2024-03-10T00:55:24.946 DEBUG:teuthology.orchestra.run.smithi106:> sudo systemctl stop ceph-75c37498-de77-11ee-95c7-87774f69a715@rgw.foo.a 2024-03-10T00:55:25.243 INFO:journalctl@ceph.rgw.foo.a.smithi106.stdout:Mar 10 00:55:24 smithi106 systemd[1]: Stopping Ceph rgw.foo.a for 75c37498-de77-11ee-95c7-87774f69a715... 2024-03-10T00:55:25.497 INFO:journalctl@ceph.rgw.foo.a.smithi106.stdout:Mar 10 00:55:25 smithi106 ceph-75c37498-de77-11ee-95c7-87774f69a715-rgw-foo-a[54939]: 2024-03-10T00:55:25.245+0000 7f95b1573700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/radosgw -n client.rgw.foo.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-10T00:55:25.497 INFO:journalctl@ceph.rgw.foo.a.smithi106.stdout:Mar 10 00:55:25 smithi106 ceph-75c37498-de77-11ee-95c7-87774f69a715-rgw-foo-a[54939]: 2024-03-10T00:55:25.245+0000 7f95bfdebac0 -1 shutting down 2024-03-10T00:55:29.469 INFO:journalctl@ceph.prometheus.a.smithi170.stdout:Mar 10 00:55:29 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-prometheus-a[52583]: ts=2024-03-10T00:55:29.079Z caller=refresh.go:99 level=error component="discovery manager scrape" discovery=http config=ceph-exporter msg="Unable to refresh target groups" err="Get \"http://172.21.15.106:8765/sd/prometheus/sd-config?service=ceph-exporter\": dial tcp 172.21.15.106:8765: connect: connection refused" 2024-03-10T00:55:29.469 INFO:journalctl@ceph.prometheus.a.smithi170.stdout:Mar 10 00:55:29 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-prometheus-a[52583]: ts=2024-03-10T00:55:29.079Z caller=refresh.go:99 level=error component="discovery manager scrape" discovery=http config=ceph msg="Unable to refresh target groups" err="Get \"http://172.21.15.106:8765/sd/prometheus/sd-config?service=mgr-prometheus\": dial tcp 172.21.15.106:8765: connect: connection refused" 2024-03-10T00:55:29.469 INFO:journalctl@ceph.prometheus.a.smithi170.stdout:Mar 10 00:55:29 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-prometheus-a[52583]: ts=2024-03-10T00:55:29.080Z caller=refresh.go:99 level=error component="discovery manager scrape" discovery=http config=nvmeof msg="Unable to refresh target groups" err="Get \"http://172.21.15.106:8765/sd/prometheus/sd-config?service=nvmeof\": dial tcp 172.21.15.106:8765: connect: connection refused" 2024-03-10T00:55:35.583 INFO:journalctl@ceph.rgw.foo.a.smithi106.stdout:Mar 10 00:55:35 smithi106 bash[80988]: time="2024-03-10T00:55:35Z" level=warning msg="StopSignal SIGTERM failed to stop container ceph-75c37498-de77-11ee-95c7-87774f69a715-rgw-foo-a in 10 seconds, resorting to SIGKILL" 2024-03-10T00:55:35.583 INFO:journalctl@ceph.rgw.foo.a.smithi106.stdout:Mar 10 00:55:35 smithi106 podman[80988]: 2024-03-10 00:55:35.266819319 +0000 UTC m=+10.125166932 container died d9af1c13f4609b8d86c21d60d00fc35f34453e04bf1b09dd4ea1162b95dbaa4f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bc4a24690ce5d7255d490e21058d8c114b55e1b93a3d8345f90b6700ad29dada, name=ceph-75c37498-de77-11ee-95c7-87774f69a715-rgw-foo-a, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.build-date=20240222, GIT_CLEAN=True, ceph=True, RELEASE=wip-yuri4-testing-2024-03-05-0854-831bc18, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-10T00:55:35.838 INFO:journalctl@ceph.rgw.foo.a.smithi106.stdout:Mar 10 00:55:35 smithi106 podman[80988]: 2024-03-10 00:55:35.676283524 +0000 UTC m=+10.534631136 container cleanup d9af1c13f4609b8d86c21d60d00fc35f34453e04bf1b09dd4ea1162b95dbaa4f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bc4a24690ce5d7255d490e21058d8c114b55e1b93a3d8345f90b6700ad29dada, name=ceph-75c37498-de77-11ee-95c7-87774f69a715-rgw-foo-a, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240222, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, ceph=True, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, RELEASE=wip-yuri4-testing-2024-03-05-0854-831bc18) 2024-03-10T00:55:35.838 INFO:journalctl@ceph.rgw.foo.a.smithi106.stdout:Mar 10 00:55:35 smithi106 bash[80988]: ceph-75c37498-de77-11ee-95c7-87774f69a715-rgw-foo-a 2024-03-10T00:55:36.050 DEBUG:teuthology.orchestra.run.smithi106:> sudo pkill -f 'journalctl -f -n 0 -u ceph-75c37498-de77-11ee-95c7-87774f69a715@rgw.foo.a.service' 2024-03-10T00:55:36.090 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-10T00:55:36.090 INFO:tasks.cephadm.ceph.rgw.foo.a:Stopped rgw.foo.a 2024-03-10T00:55:36.090 INFO:tasks.cephadm.prometheus.a:Stopping prometheus.a... 2024-03-10T00:55:36.090 DEBUG:teuthology.orchestra.run.smithi170:> sudo systemctl stop ceph-75c37498-de77-11ee-95c7-87774f69a715@prometheus.a 2024-03-10T00:55:36.406 INFO:journalctl@ceph.prometheus.a.smithi170.stdout:Mar 10 00:55:36 smithi170 systemd[1]: Stopping Ceph prometheus.a for 75c37498-de77-11ee-95c7-87774f69a715... 2024-03-10T00:55:36.679 INFO:journalctl@ceph.prometheus.a.smithi170.stdout:Mar 10 00:55:36 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-prometheus-a[52583]: ts=2024-03-10T00:55:36.408Z caller=main.go:840 level=warn msg="Received SIGTERM, exiting gracefully..." 2024-03-10T00:55:36.679 INFO:journalctl@ceph.prometheus.a.smithi170.stdout:Mar 10 00:55:36 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-prometheus-a[52583]: ts=2024-03-10T00:55:36.408Z caller=main.go:864 level=info msg="Stopping scrape discovery manager..." 2024-03-10T00:55:36.679 INFO:journalctl@ceph.prometheus.a.smithi170.stdout:Mar 10 00:55:36 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-prometheus-a[52583]: ts=2024-03-10T00:55:36.408Z caller=main.go:878 level=info msg="Stopping notify discovery manager..." 2024-03-10T00:55:36.680 INFO:journalctl@ceph.prometheus.a.smithi170.stdout:Mar 10 00:55:36 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-prometheus-a[52583]: ts=2024-03-10T00:55:36.408Z caller=main.go:860 level=info msg="Scrape discovery manager stopped" 2024-03-10T00:55:36.680 INFO:journalctl@ceph.prometheus.a.smithi170.stdout:Mar 10 00:55:36 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-prometheus-a[52583]: ts=2024-03-10T00:55:36.408Z caller=manager.go:988 level=info component="rule manager" msg="Stopping rule manager..." 2024-03-10T00:55:36.680 INFO:journalctl@ceph.prometheus.a.smithi170.stdout:Mar 10 00:55:36 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-prometheus-a[52583]: ts=2024-03-10T00:55:36.408Z caller=manager.go:998 level=info component="rule manager" msg="Rule manager stopped" 2024-03-10T00:55:36.680 INFO:journalctl@ceph.prometheus.a.smithi170.stdout:Mar 10 00:55:36 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-prometheus-a[52583]: ts=2024-03-10T00:55:36.408Z caller=main.go:915 level=info msg="Stopping scrape manager..." 2024-03-10T00:55:36.680 INFO:journalctl@ceph.prometheus.a.smithi170.stdout:Mar 10 00:55:36 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-prometheus-a[52583]: ts=2024-03-10T00:55:36.408Z caller=main.go:874 level=info msg="Notify discovery manager stopped" 2024-03-10T00:55:36.680 INFO:journalctl@ceph.prometheus.a.smithi170.stdout:Mar 10 00:55:36 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-prometheus-a[52583]: ts=2024-03-10T00:55:36.408Z caller=main.go:907 level=info msg="Scrape manager stopped" 2024-03-10T00:55:36.680 INFO:journalctl@ceph.prometheus.a.smithi170.stdout:Mar 10 00:55:36 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-prometheus-a[52583]: ts=2024-03-10T00:55:36.409Z caller=notifier.go:606 level=info component=notifier msg="Stopping notification manager..." 2024-03-10T00:55:36.680 INFO:journalctl@ceph.prometheus.a.smithi170.stdout:Mar 10 00:55:36 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-prometheus-a[52583]: ts=2024-03-10T00:55:36.409Z caller=main.go:1135 level=info msg="Notifier manager stopped" 2024-03-10T00:55:36.680 INFO:journalctl@ceph.prometheus.a.smithi170.stdout:Mar 10 00:55:36 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-prometheus-a[52583]: ts=2024-03-10T00:55:36.409Z caller=main.go:1147 level=info msg="See you next time!" 2024-03-10T00:55:36.680 INFO:journalctl@ceph.prometheus.a.smithi170.stdout:Mar 10 00:55:36 smithi170 podman[68942]: 2024-03-10 00:55:36.419522883 +0000 UTC m=+0.120650632 container died 13aadf83806a9cd335ab57b8cd34ded81c4876c5a3cdd8bf6ce9cb310c1b4de1 (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-75c37498-de77-11ee-95c7-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-10T00:55:37.245 INFO:journalctl@ceph.prometheus.a.smithi170.stdout:Mar 10 00:55:36 smithi170 podman[68942]: 2024-03-10 00:55:36.9810902 +0000 UTC m=+0.682217993 container cleanup 13aadf83806a9cd335ab57b8cd34ded81c4876c5a3cdd8bf6ce9cb310c1b4de1 (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-75c37498-de77-11ee-95c7-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-10T00:55:37.246 INFO:journalctl@ceph.prometheus.a.smithi170.stdout:Mar 10 00:55:36 smithi170 bash[68942]: ceph-75c37498-de77-11ee-95c7-87774f69a715-prometheus-a 2024-03-10T00:55:37.398 DEBUG:teuthology.orchestra.run.smithi170:> sudo pkill -f 'journalctl -f -n 0 -u ceph-75c37498-de77-11ee-95c7-87774f69a715@prometheus.a.service' 2024-03-10T00:55:37.436 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-10T00:55:37.437 INFO:tasks.cephadm.prometheus.a:Stopped prometheus.a 2024-03-10T00:55:37.437 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 75c37498-de77-11ee-95c7-87774f69a715 --force --keep-logs 2024-03-10T00:55:37.633 INFO:teuthology.orchestra.run.smithi106.stdout:Deleting cluster with fsid: 75c37498-de77-11ee-95c7-87774f69a715 2024-03-10T00:55:39.588 INFO:journalctl@ceph.alertmanager.a.smithi106.stdout:Mar 10 00:55:39 smithi106 systemd[1]: Stopping Ceph alertmanager.a for 75c37498-de77-11ee-95c7-87774f69a715... 2024-03-10T00:55:40.037 INFO:journalctl@ceph.alertmanager.a.smithi106.stdout:Mar 10 00:55:39 smithi106 ceph-75c37498-de77-11ee-95c7-87774f69a715-alertmanager-a[65783]: ts=2024-03-10T00:55:39.775Z caller=main.go:583 level=info msg="Received SIGTERM, exiting gracefully..." 2024-03-10T00:55:40.037 INFO:journalctl@ceph.alertmanager.a.smithi106.stdout:Mar 10 00:55:39 smithi106 podman[82344]: 2024-03-10 00:55:39.796627996 +0000 UTC m=+0.122911028 container died d5b698eacae7ffd2e524a0b4c908c8a8efc22ce9c1628ee6ea49ffbca227446b (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-75c37498-de77-11ee-95c7-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-03-10T00:55:40.338 INFO:journalctl@ceph.alertmanager.a.smithi106.stdout:Mar 10 00:55:40 smithi106 podman[82344]: 2024-03-10 00:55:40.214922951 +0000 UTC m=+0.541205986 container cleanup d5b698eacae7ffd2e524a0b4c908c8a8efc22ce9c1628ee6ea49ffbca227446b (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-75c37498-de77-11ee-95c7-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-03-10T00:55:40.338 INFO:journalctl@ceph.alertmanager.a.smithi106.stdout:Mar 10 00:55:40 smithi106 bash[82344]: ceph-75c37498-de77-11ee-95c7-87774f69a715-alertmanager-a 2024-03-10T00:55:40.752 INFO:journalctl@ceph.alertmanager.a.smithi106.stdout:Mar 10 00:55:40 smithi106 podman[82363]: 2024-03-10 00:55:40.432679374 +0000 UTC m=+0.639107136 container remove d5b698eacae7ffd2e524a0b4c908c8a8efc22ce9c1628ee6ea49ffbca227446b (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-75c37498-de77-11ee-95c7-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-03-10T00:55:40.752 INFO:journalctl@ceph.alertmanager.a.smithi106.stdout:Mar 10 00:55:40 smithi106 podman[82363]: 2024-03-10 00:55:40.516612366 +0000 UTC m=+0.723040125 volume remove 31471fbefba53045ac0dfe2604d04ad0894fba8896fd801af5dc0230cda0b16d 2024-03-10T00:55:41.087 INFO:journalctl@ceph.node-exporter.a.smithi106.stdout:Mar 10 00:55:40 smithi106 systemd[1]: Stopping Ceph node-exporter.a for 75c37498-de77-11ee-95c7-87774f69a715... 2024-03-10T00:55:41.088 INFO:journalctl@ceph.alertmanager.a.smithi106.stdout:Mar 10 00:55:40 smithi106 systemd[1]: ceph-75c37498-de77-11ee-95c7-87774f69a715@alertmanager.a.service: Deactivated successfully. 2024-03-10T00:55:41.088 INFO:journalctl@ceph.alertmanager.a.smithi106.stdout:Mar 10 00:55:40 smithi106 systemd[1]: Stopped Ceph alertmanager.a for 75c37498-de77-11ee-95c7-87774f69a715. 2024-03-10T00:55:41.088 INFO:journalctl@ceph.alertmanager.a.smithi106.stdout:Mar 10 00:55:40 smithi106 systemd[1]: ceph-75c37498-de77-11ee-95c7-87774f69a715@alertmanager.a.service: Consumed 1.202s CPU time. 2024-03-10T00:55:41.500 INFO:journalctl@ceph.node-exporter.a.smithi106.stdout:Mar 10 00:55:41 smithi106 podman[82479]: 2024-03-10 00:55:41.233008757 +0000 UTC m=+0.110551880 container died fdb97a00c98ec0b681c0342dbd819bc2663f981caccbdb3dfd380c352759bdb7 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-75c37498-de77-11ee-95c7-87774f69a715-node-exporter-a, maintainer=The Prometheus Authors ) 2024-03-10T00:55:41.838 INFO:journalctl@ceph.node-exporter.a.smithi106.stdout:Mar 10 00:55:41 smithi106 podman[82479]: 2024-03-10 00:55:41.602788246 +0000 UTC m=+0.480331370 container cleanup fdb97a00c98ec0b681c0342dbd819bc2663f981caccbdb3dfd380c352759bdb7 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-75c37498-de77-11ee-95c7-87774f69a715-node-exporter-a, maintainer=The Prometheus Authors ) 2024-03-10T00:55:41.838 INFO:journalctl@ceph.node-exporter.a.smithi106.stdout:Mar 10 00:55:41 smithi106 bash[82479]: ceph-75c37498-de77-11ee-95c7-87774f69a715-node-exporter-a 2024-03-10T00:55:42.307 INFO:journalctl@ceph.node-exporter.a.smithi106.stdout:Mar 10 00:55:41 smithi106 podman[82499]: 2024-03-10 00:55:41.845601565 +0000 UTC m=+0.615576267 container remove fdb97a00c98ec0b681c0342dbd819bc2663f981caccbdb3dfd380c352759bdb7 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-75c37498-de77-11ee-95c7-87774f69a715-node-exporter-a, maintainer=The Prometheus Authors ) 2024-03-10T00:55:42.307 INFO:journalctl@ceph.node-exporter.a.smithi106.stdout:Mar 10 00:55:41 smithi106 systemd[1]: ceph-75c37498-de77-11ee-95c7-87774f69a715@node-exporter.a.service: Main process exited, code=exited, status=143/n/a 2024-03-10T00:55:42.307 INFO:journalctl@ceph.node-exporter.a.smithi106.stdout:Mar 10 00:55:42 smithi106 systemd[1]: ceph-75c37498-de77-11ee-95c7-87774f69a715@node-exporter.a.service: Failed with result 'exit-code'. 2024-03-10T00:55:42.307 INFO:journalctl@ceph.node-exporter.a.smithi106.stdout:Mar 10 00:55:42 smithi106 systemd[1]: Stopped Ceph node-exporter.a for 75c37498-de77-11ee-95c7-87774f69a715. 2024-03-10T00:55:42.307 INFO:journalctl@ceph.node-exporter.a.smithi106.stdout:Mar 10 00:55:42 smithi106 systemd[1]: ceph-75c37498-de77-11ee-95c7-87774f69a715@node-exporter.a.service: Consumed 1.954s CPU time. 2024-03-10T00:55:42.692 DEBUG:teuthology.orchestra.run.smithi170:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 75c37498-de77-11ee-95c7-87774f69a715 --force --keep-logs 2024-03-10T00:55:42.883 INFO:teuthology.orchestra.run.smithi170.stdout:Deleting cluster with fsid: 75c37498-de77-11ee-95c7-87774f69a715 2024-03-10T00:55:44.524 INFO:journalctl@ceph.iscsi.iscsi.a.smithi170.stdout:Mar 10 00:55:44 smithi170 systemd[1]: Stopping Ceph iscsi.iscsi.a for 75c37498-de77-11ee-95c7-87774f69a715... 2024-03-10T00:55:45.024 INFO:journalctl@ceph.iscsi.iscsi.a.smithi170.stdout:Mar 10 00:55:44 smithi170 ceph-75c37498-de77-11ee-95c7-87774f69a715-iscsi-iscsi-a[50134]: debug Shutdown received 2024-03-10T00:55:54.950 INFO:journalctl@ceph.iscsi.iscsi.a.smithi170.stdout:Mar 10 00:55:54 smithi170 bash[69402]: time="2024-03-10T00:55:54Z" level=warning msg="StopSignal SIGTERM failed to stop container ceph-75c37498-de77-11ee-95c7-87774f69a715-iscsi-iscsi-a in 10 seconds, resorting to SIGKILL" 2024-03-10T00:55:54.950 INFO:journalctl@ceph.iscsi.iscsi.a.smithi170.stdout:Mar 10 00:55:54 smithi170 podman[69402]: 2024-03-10 00:55:54.693575481 +0000 UTC m=+10.131573192 container died 8e6ee66d69a226ced52ef8f21ac185a1b42eb4a624c9c2d7e54f9757bfce2463 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bc4a24690ce5d7255d490e21058d8c114b55e1b93a3d8345f90b6700ad29dada, name=ceph-75c37498-de77-11ee-95c7-87774f69a715-iscsi-iscsi-a, RELEASE=wip-yuri4-testing-2024-03-05-0854-831bc18, maintainer=Guillaume Abrioux , ceph=True, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.build-date=20240222, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-10T00:55:55.274 INFO:journalctl@ceph.iscsi.iscsi.a.smithi170.stdout:Mar 10 00:55:55 smithi170 podman[69402]: 2024-03-10 00:55:55.118291464 +0000 UTC m=+10.556289189 container cleanup 8e6ee66d69a226ced52ef8f21ac185a1b42eb4a624c9c2d7e54f9757bfce2463 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bc4a24690ce5d7255d490e21058d8c114b55e1b93a3d8345f90b6700ad29dada, name=ceph-75c37498-de77-11ee-95c7-87774f69a715-iscsi-iscsi-a, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-yuri4-testing-2024-03-05-0854-831bc18, ceph=True, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240222, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux ) 2024-03-10T00:55:55.274 INFO:journalctl@ceph.iscsi.iscsi.a.smithi170.stdout:Mar 10 00:55:55 smithi170 bash[69402]: ceph-75c37498-de77-11ee-95c7-87774f69a715-iscsi-iscsi-a 2024-03-10T00:55:55.558 INFO:journalctl@ceph.iscsi.iscsi.a.smithi170.stdout:Mar 10 00:55:55 smithi170 podman[69439]: 2024-03-10 00:55:55.302898997 +0000 UTC m=+0.610254153 container remove 8e6ee66d69a226ced52ef8f21ac185a1b42eb4a624c9c2d7e54f9757bfce2463 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bc4a24690ce5d7255d490e21058d8c114b55e1b93a3d8345f90b6700ad29dada, name=ceph-75c37498-de77-11ee-95c7-87774f69a715-iscsi-iscsi-a, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.vendor=CentOS, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_CLEAN=True, RELEASE=wip-yuri4-testing-2024-03-05-0854-831bc18, org.label-schema.build-date=20240222) 2024-03-10T00:55:55.558 INFO:journalctl@ceph.iscsi.iscsi.a.smithi170.stdout:Mar 10 00:55:55 smithi170 systemd[1]: ceph-75c37498-de77-11ee-95c7-87774f69a715@iscsi.iscsi.a.service: Main process exited, code=exited, status=137/n/a 2024-03-10T00:55:55.558 INFO:journalctl@ceph.iscsi.iscsi.a.smithi170.stdout:Mar 10 00:55:55 smithi170 systemd[1]: ceph-75c37498-de77-11ee-95c7-87774f69a715@iscsi.iscsi.a.service: Failed with result 'exit-code'. 2024-03-10T00:55:55.558 INFO:journalctl@ceph.iscsi.iscsi.a.smithi170.stdout:Mar 10 00:55:55 smithi170 systemd[1]: Stopped Ceph iscsi.iscsi.a for 75c37498-de77-11ee-95c7-87774f69a715. 2024-03-10T00:55:55.558 INFO:journalctl@ceph.iscsi.iscsi.a.smithi170.stdout:Mar 10 00:55:55 smithi170 systemd[1]: ceph-75c37498-de77-11ee-95c7-87774f69a715@iscsi.iscsi.a.service: Consumed 2.867s CPU time. 2024-03-10T00:55:57.273 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Mar 10 00:55:57 smithi170 systemd[1]: Stopping Ceph grafana.a for 75c37498-de77-11ee-95c7-87774f69a715... 2024-03-10T00:55:57.641 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Mar 10 00:55:57 smithi170 podman[69718]: 2024-03-10 00:55:57.38167546 +0000 UTC m=+0.114317123 container died 247603ac339fcf8e08b1a4e78f010d652735b920717e12700166512af49899fe (image=quay.io/ceph/ceph-grafana:9.4.12, name=ceph-75c37498-de77-11ee-95c7-87774f69a715-grafana-a, io.buildah.version=1.32.0, vendor=Red Hat, Inc., com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, io.openshift.tags=base rhel8, com.redhat.component=ubi8-container, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, name=ubi8, description=Ceph Grafana Container, version=8.5, com.redhat.build-host=cpt-1007.osbs.prod.upshift.rdu2.redhat.com, io.openshift.expose-services=, release=239.1651231664, build-date=2022-04-29T11:48:11.055897, maintainer=Paul Cuzner , distribution-scope=public, io.k8s.display-name=Red Hat Universal Base Image 8, summary=Grafana Container configured for Ceph mgr/dashboard integration, architecture=x86_64, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-239.1651231664, vcs-type=git) 2024-03-10T00:55:57.999 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Mar 10 00:55:57 smithi170 podman[69718]: 2024-03-10 00:55:57.81014883 +0000 UTC m=+0.542790499 container cleanup 247603ac339fcf8e08b1a4e78f010d652735b920717e12700166512af49899fe (image=quay.io/ceph/ceph-grafana:9.4.12, name=ceph-75c37498-de77-11ee-95c7-87774f69a715-grafana-a, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, release=239.1651231664, distribution-scope=public, summary=Grafana Container configured for Ceph mgr/dashboard integration, vendor=Red Hat, Inc., name=ubi8, architecture=x86_64, io.openshift.expose-services=, io.openshift.tags=base rhel8, com.redhat.component=ubi8-container, build-date=2022-04-29T11:48:11.055897, maintainer=Paul Cuzner , vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-239.1651231664, io.buildah.version=1.32.0, com.redhat.build-host=cpt-1007.osbs.prod.upshift.rdu2.redhat.com, description=Ceph Grafana Container, vcs-type=git, io.k8s.display-name=Red Hat Universal Base Image 8, version=8.5) 2024-03-10T00:55:58.000 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Mar 10 00:55:57 smithi170 bash[69718]: ceph-75c37498-de77-11ee-95c7-87774f69a715-grafana-a 2024-03-10T00:55:58.274 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Mar 10 00:55:58 smithi170 podman[69737]: 2024-03-10 00:55:58.002988575 +0000 UTC m=+0.621768905 container remove 247603ac339fcf8e08b1a4e78f010d652735b920717e12700166512af49899fe (image=quay.io/ceph/ceph-grafana:9.4.12, name=ceph-75c37498-de77-11ee-95c7-87774f69a715-grafana-a, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, summary=Grafana Container configured for Ceph mgr/dashboard integration, distribution-scope=public, version=8.5, name=ubi8, build-date=2022-04-29T11:48:11.055897, io.k8s.display-name=Red Hat Universal Base Image 8, com.redhat.build-host=cpt-1007.osbs.prod.upshift.rdu2.redhat.com, maintainer=Paul Cuzner , vcs-type=git, com.redhat.component=ubi8-container, vendor=Red Hat, Inc., description=Ceph Grafana Container, release=239.1651231664, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.openshift.tags=base rhel8, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-239.1651231664, architecture=x86_64, io.buildah.version=1.32.0, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, io.openshift.expose-services=) 2024-03-10T00:55:58.274 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Mar 10 00:55:58 smithi170 systemd[1]: ceph-75c37498-de77-11ee-95c7-87774f69a715@grafana.a.service: Main process exited, code=exited, status=143/n/a 2024-03-10T00:55:58.274 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Mar 10 00:55:58 smithi170 systemd[1]: ceph-75c37498-de77-11ee-95c7-87774f69a715@grafana.a.service: Failed with result 'exit-code'. 2024-03-10T00:55:58.274 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Mar 10 00:55:58 smithi170 systemd[1]: Stopped Ceph grafana.a for 75c37498-de77-11ee-95c7-87774f69a715. 2024-03-10T00:55:58.274 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Mar 10 00:55:58 smithi170 systemd[1]: ceph-75c37498-de77-11ee-95c7-87774f69a715@grafana.a.service: Consumed 4.745s CPU time. 2024-03-10T00:55:58.774 INFO:journalctl@ceph.node-exporter.b.smithi170.stdout:Mar 10 00:55:58 smithi170 systemd[1]: Stopping Ceph node-exporter.b for 75c37498-de77-11ee-95c7-87774f69a715... 2024-03-10T00:55:58.774 INFO:journalctl@ceph.node-exporter.b.smithi170.stdout:Mar 10 00:55:58 smithi170 podman[69853]: 2024-03-10 00:55:58.668794027 +0000 UTC m=+0.104455502 container died 5b7019120d1fa43fb65545e472c525a122a52c3005e11fb93201070a4cb57cbf (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-75c37498-de77-11ee-95c7-87774f69a715-node-exporter-b, maintainer=The Prometheus Authors ) 2024-03-10T00:55:59.273 INFO:journalctl@ceph.node-exporter.b.smithi170.stdout:Mar 10 00:55:59 smithi170 podman[69853]: 2024-03-10 00:55:59.088959476 +0000 UTC m=+0.524620953 container cleanup 5b7019120d1fa43fb65545e472c525a122a52c3005e11fb93201070a4cb57cbf (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-75c37498-de77-11ee-95c7-87774f69a715-node-exporter-b, maintainer=The Prometheus Authors ) 2024-03-10T00:55:59.274 INFO:journalctl@ceph.node-exporter.b.smithi170.stdout:Mar 10 00:55:59 smithi170 bash[69853]: ceph-75c37498-de77-11ee-95c7-87774f69a715-node-exporter-b 2024-03-10T00:55:59.685 INFO:journalctl@ceph.node-exporter.b.smithi170.stdout:Mar 10 00:55:59 smithi170 podman[69872]: 2024-03-10 00:55:59.294144309 +0000 UTC m=+0.630423989 container remove 5b7019120d1fa43fb65545e472c525a122a52c3005e11fb93201070a4cb57cbf (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-75c37498-de77-11ee-95c7-87774f69a715-node-exporter-b, maintainer=The Prometheus Authors ) 2024-03-10T00:55:59.685 INFO:journalctl@ceph.node-exporter.b.smithi170.stdout:Mar 10 00:55:59 smithi170 systemd[1]: ceph-75c37498-de77-11ee-95c7-87774f69a715@node-exporter.b.service: Main process exited, code=exited, status=143/n/a 2024-03-10T00:55:59.685 INFO:journalctl@ceph.node-exporter.b.smithi170.stdout:Mar 10 00:55:59 smithi170 systemd[1]: ceph-75c37498-de77-11ee-95c7-87774f69a715@node-exporter.b.service: Failed with result 'exit-code'. 2024-03-10T00:55:59.686 INFO:journalctl@ceph.node-exporter.b.smithi170.stdout:Mar 10 00:55:59 smithi170 systemd[1]: Stopped Ceph node-exporter.b for 75c37498-de77-11ee-95c7-87774f69a715. 2024-03-10T00:55:59.686 INFO:journalctl@ceph.node-exporter.b.smithi170.stdout:Mar 10 00:55:59 smithi170 systemd[1]: ceph-75c37498-de77-11ee-95c7-87774f69a715@node-exporter.b.service: Consumed 1.800s CPU time. 2024-03-10T00:56:00.160 DEBUG:teuthology.orchestra.run.smithi106:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-03-10T00:56:00.195 DEBUG:teuthology.orchestra.run.smithi170:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-03-10T00:56:00.224 INFO:tasks.cephadm:Archiving crash dumps... 2024-03-10T00:56:00.226 DEBUG:teuthology.misc:Transferring archived files from smithi106:/var/lib/ceph/75c37498-de77-11ee-95c7-87774f69a715/crash to /home/teuthworker/archive/yuriw-2024-03-08_16:20:46-rados-wip-yuri4-testing-2024-03-05-0854-distro-default-smithi/7587708/remote/smithi106/crash 2024-03-10T00:56:00.226 DEBUG:teuthology.orchestra.run.smithi106:> sudo tar c -f - -C /var/lib/ceph/75c37498-de77-11ee-95c7-87774f69a715/crash -- . 2024-03-10T00:56:00.274 INFO:teuthology.orchestra.run.smithi106.stderr:tar: /var/lib/ceph/75c37498-de77-11ee-95c7-87774f69a715/crash: Cannot open: No such file or directory 2024-03-10T00:56:00.274 INFO:teuthology.orchestra.run.smithi106.stderr:tar: Error is not recoverable: exiting now 2024-03-10T00:56:00.276 DEBUG:teuthology.misc:Transferring archived files from smithi170:/var/lib/ceph/75c37498-de77-11ee-95c7-87774f69a715/crash to /home/teuthworker/archive/yuriw-2024-03-08_16:20:46-rados-wip-yuri4-testing-2024-03-05-0854-distro-default-smithi/7587708/remote/smithi170/crash 2024-03-10T00:56:00.276 DEBUG:teuthology.orchestra.run.smithi170:> sudo tar c -f - -C /var/lib/ceph/75c37498-de77-11ee-95c7-87774f69a715/crash -- . 2024-03-10T00:56:00.313 INFO:teuthology.orchestra.run.smithi170.stderr:tar: /var/lib/ceph/75c37498-de77-11ee-95c7-87774f69a715/crash: Cannot open: No such file or directory 2024-03-10T00:56:00.314 INFO:teuthology.orchestra.run.smithi170.stderr:tar: Error is not recoverable: exiting now 2024-03-10T00:56:00.314 INFO:tasks.cephadm:Checking cluster log for badness... 2024-03-10T00:56:00.315 DEBUG:teuthology.orchestra.run.smithi106:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/75c37498-de77-11ee-95c7-87774f69a715/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v MON_DOWN | head -n 1 2024-03-10T00:56:00.344 INFO:teuthology.orchestra.run.smithi106.stdout:2024-03-10T00:50:47.468873+0000 mon.a (mon.0) 648 : cluster [WRN] Health check failed: 1 stray daemon(s) not managed by cephadm (CEPHADM_STRAY_DAEMON) 2024-03-10T00:56:00.345 WARNING:tasks.cephadm:Found errors (ERR|WRN|SEC) in cluster log 2024-03-10T00:56:00.345 DEBUG:teuthology.orchestra.run.smithi106:> sudo egrep '\[SEC\]' /var/log/ceph/75c37498-de77-11ee-95c7-87774f69a715/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v MON_DOWN | head -n 1 2024-03-10T00:56:00.413 DEBUG:teuthology.orchestra.run.smithi106:> sudo egrep '\[ERR\]' /var/log/ceph/75c37498-de77-11ee-95c7-87774f69a715/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v MON_DOWN | head -n 1 2024-03-10T00:56:00.482 DEBUG:teuthology.orchestra.run.smithi106:> sudo egrep '\[WRN\]' /var/log/ceph/75c37498-de77-11ee-95c7-87774f69a715/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v MON_DOWN | head -n 1 2024-03-10T00:56:00.551 INFO:teuthology.orchestra.run.smithi106.stdout:2024-03-10T00:50:47.468873+0000 mon.a (mon.0) 648 : cluster [WRN] Health check failed: 1 stray daemon(s) not managed by cephadm (CEPHADM_STRAY_DAEMON) 2024-03-10T00:56:00.552 INFO:tasks.cephadm:Compressing logs... 2024-03-10T00:56:00.552 DEBUG:teuthology.orchestra.run.smithi106:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-03-10T00:56:00.593 DEBUG:teuthology.orchestra.run.smithi170:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-03-10T00:56:00.618 INFO:teuthology.orchestra.run.smithi106.stderr:find: ‘/var/log/rbd-target-api’: No such file or directory 2024-03-10T00:56:00.619 INFO:teuthology.orchestra.run.smithi106.stderr:gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-03-10T00:56:00.622 INFO:teuthology.orchestra.run.smithi106.stderr:gzip -5 --verbose -- /var/log/ceph/75c37498-de77-11ee-95c7-87774f69a715/ceph-mon.a.log 2024-03-10T00:56:00.622 INFO:teuthology.orchestra.run.smithi106.stderr:gzip/var/log/ceph/cephadm.log: -5 --verbose -- /var/log/ceph/75c37498-de77-11ee-95c7-87774f69a715/ceph.audit.log 2024-03-10T00:56:00.622 INFO:teuthology.orchestra.run.smithi106.stderr:/var/log/ceph/75c37498-de77-11ee-95c7-87774f69a715/ceph-mon.a.log: gzip -5 --verbose -- /var/log/ceph/75c37498-de77-11ee-95c7-87774f69a715/ceph-client.rgw.foo.a.log 2024-03-10T00:56:00.623 INFO:teuthology.orchestra.run.smithi106.stderr:/var/log/ceph/75c37498-de77-11ee-95c7-87774f69a715/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/75c37498-de77-11ee-95c7-87774f69a715/ceph-osd.1.log 2024-03-10T00:56:00.623 INFO:teuthology.orchestra.run.smithi106.stderr:/var/log/ceph/75c37498-de77-11ee-95c7-87774f69a715/ceph-client.rgw.foo.a.log: gzip -5 --verbose -- /var/log/ceph/75c37498-de77-11ee-95c7-87774f69a715/ceph-osd.3.log 2024-03-10T00:56:00.623 INFO:teuthology.orchestra.run.smithi106.stderr: 57.7% -- replaced with /var/log/ceph/75c37498-de77-11ee-95c7-87774f69a715/ceph-client.rgw.foo.a.log.gz 2024-03-10T00:56:00.623 INFO:teuthology.orchestra.run.smithi106.stderr:/var/log/ceph/75c37498-de77-11ee-95c7-87774f69a715/ceph-osd.1.log: gzip -5 --verbose -- /var/log/ceph/75c37498-de77-11ee-95c7-87774f69a715/ceph-mgr.y.log 2024-03-10T00:56:00.623 INFO:teuthology.orchestra.run.smithi106.stderr:/var/log/ceph/75c37498-de77-11ee-95c7-87774f69a715/ceph-osd.3.log: gzip -5 --verbose -- /var/log/ceph/75c37498-de77-11ee-95c7-87774f69a715/ceph-osd.0.log 2024-03-10T00:56:00.629 INFO:teuthology.orchestra.run.smithi170.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose --: No such file or directory /var/log/ceph/cephadm.log 2024-03-10T00:56:00.629 INFO:teuthology.orchestra.run.smithi170.stderr: 2024-03-10T00:56:00.629 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/75c37498-de77-11ee-95c7-87774f69a715/ceph-osd.6.log 2024-03-10T00:56:00.629 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cephadm.log: gzip -5 --verbose -- /var/log/ceph/75c37498-de77-11ee-95c7-87774f69a715/ceph.audit.log 2024-03-10T00:56:00.629 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/75c37498-de77-11ee-95c7-87774f69a715/ceph-osd.6.log: gzip -5 --verbose -- /var/log/ceph/75c37498-de77-11ee-95c7-87774f69a715/ceph-osd.4.log 2024-03-10T00:56:00.629 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/75c37498-de77-11ee-95c7-87774f69a715/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/75c37498-de77-11ee-95c7-87774f69a715/ceph-osd.7.log 2024-03-10T00:56:00.629 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/75c37498-de77-11ee-95c7-87774f69a715/ceph-osd.5.log 2024-03-10T00:56:00.629 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/75c37498-de77-11ee-95c7-87774f69a715/ceph-osd.4.log: /var/log/ceph/75c37498-de77-11ee-95c7-87774f69a715/ceph-osd.7.log: gzip -5 --verbose -- /var/log/ceph/75c37498-de77-11ee-95c7-87774f69a715/ceph-mgr.x.log 2024-03-10T00:56:00.629 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/75c37498-de77-11ee-95c7-87774f69a715/ceph-osd.5.log: gzip -5 --verbose -- /var/log/ceph/75c37498-de77-11ee-95c7-87774f69a715/tcmu-runner.log 2024-03-10T00:56:00.629 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/75c37498-de77-11ee-95c7-87774f69a715/ceph-mgr.x.log: gzip -5 --verbose -- /var/log/ceph/75c37498-de77-11ee-95c7-87774f69a715/ceph.cephadm.log 2024-03-10T00:56:00.629 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/75c37498-de77-11ee-95c7-87774f69a715/tcmu-runner.log: 90.5% -- replaced with /var/log/ceph/75c37498-de77-11ee-95c7-87774f69a715/ceph.audit.log.gz 2024-03-10T00:56:00.629 INFO:teuthology.orchestra.run.smithi170.stderr: 67.8% -- replaced with /var/log/ceph/75c37498-de77-11ee-95c7-87774f69a715/tcmu-runner.log.gz 2024-03-10T00:56:00.629 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/75c37498-de77-11ee-95c7-87774f69a715/ceph.log 2024-03-10T00:56:00.630 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/75c37498-de77-11ee-95c7-87774f69a715/ceph.cephadm.log: gzip -5 --verbose -- /var/log/ceph/75c37498-de77-11ee-95c7-87774f69a715/ceph-volume.log 2024-03-10T00:56:00.630 INFO:teuthology.orchestra.run.smithi170.stderr: 81.5% -- replaced with /var/log/ceph/75c37498-de77-11ee-95c7-87774f69a715/ceph.cephadm.log.gz 2024-03-10T00:56:00.630 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/75c37498-de77-11ee-95c7-87774f69a715/ceph.log: gzip -5 --verbose -- /var/log/ceph/75c37498-de77-11ee-95c7-87774f69a715/ceph-mon.b.log 2024-03-10T00:56:00.630 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/75c37498-de77-11ee-95c7-87774f69a715/ceph-volume.log: 83.1% -- replaced with /var/log/ceph/cephadm.log.gz 2024-03-10T00:56:00.630 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/75c37498-de77-11ee-95c7-87774f69a715/ceph-mon.b.log: 86.0% -- replaced with /var/log/ceph/75c37498-de77-11ee-95c7-87774f69a715/ceph.log.gz 2024-03-10T00:56:00.630 INFO:teuthology.orchestra.run.smithi170.stderr: 90.7% -- replaced with /var/log/ceph/75c37498-de77-11ee-95c7-87774f69a715/ceph-mgr.x.log.gz 2024-03-10T00:56:00.634 INFO:teuthology.orchestra.run.smithi106.stderr:/var/log/ceph/75c37498-de77-11ee-95c7-87774f69a715/ceph-mgr.y.log: 94.0% -- replaced with /var/log/ceph/75c37498-de77-11ee-95c7-87774f69a715/ceph.audit.log.gz 2024-03-10T00:56:00.634 INFO:teuthology.orchestra.run.smithi106.stderr: 86.7% -- replaced with /var/log/ceph/cephadm.log.gz 2024-03-10T00:56:00.634 INFO:teuthology.orchestra.run.smithi106.stderr:gzip -5 --verbose -- /var/log/ceph/75c37498-de77-11ee-95c7-87774f69a715/ceph.cephadm.log 2024-03-10T00:56:00.634 INFO:teuthology.orchestra.run.smithi106.stderr:/var/log/ceph/75c37498-de77-11ee-95c7-87774f69a715/ceph-osd.0.log: gzip -5 --verbose -- /var/log/ceph/75c37498-de77-11ee-95c7-87774f69a715/ceph.log 2024-03-10T00:56:00.634 INFO:teuthology.orchestra.run.smithi106.stderr:/var/log/ceph/75c37498-de77-11ee-95c7-87774f69a715/ceph.cephadm.log: 89.1% -- replaced with /var/log/ceph/75c37498-de77-11ee-95c7-87774f69a715/ceph.cephadm.log.gz 2024-03-10T00:56:00.635 INFO:teuthology.orchestra.run.smithi106.stderr:gzip -5 --verbose -- /var/log/ceph/75c37498-de77-11ee-95c7-87774f69a715/ceph-osd.2.log 2024-03-10T00:56:00.635 INFO:teuthology.orchestra.run.smithi106.stderr:/var/log/ceph/75c37498-de77-11ee-95c7-87774f69a715/ceph.log: gzip -5 --verbose -- /var/log/ceph/75c37498-de77-11ee-95c7-87774f69a715/ceph-volume.log 2024-03-10T00:56:00.635 INFO:teuthology.orchestra.run.smithi106.stderr:/var/log/ceph/75c37498-de77-11ee-95c7-87774f69a715/ceph-osd.2.log: 91.9% -- replaced with /var/log/ceph/75c37498-de77-11ee-95c7-87774f69a715/ceph.log.gz 2024-03-10T00:56:00.635 INFO:teuthology.orchestra.run.smithi106.stderr:gzip -5 --verbose -- /var/log/ceph/75c37498-de77-11ee-95c7-87774f69a715/ceph-mon.c.log 2024-03-10T00:56:00.680 INFO:teuthology.orchestra.run.smithi170.stderr: 90.2% -- replaced with /var/log/ceph/75c37498-de77-11ee-95c7-87774f69a715/ceph-volume.log.gz 2024-03-10T00:56:00.725 INFO:teuthology.orchestra.run.smithi170.stderr: 92.2% -- replaced with /var/log/ceph/75c37498-de77-11ee-95c7-87774f69a715/ceph-mon.b.log.gz 2024-03-10T00:56:00.740 INFO:teuthology.orchestra.run.smithi106.stderr:/var/log/ceph/75c37498-de77-11ee-95c7-87774f69a715/ceph-volume.log: /var/log/ceph/75c37498-de77-11ee-95c7-87774f69a715/ceph-mon.c.log: 90.2% -- replaced with /var/log/ceph/75c37498-de77-11ee-95c7-87774f69a715/ceph-volume.log.gz 2024-03-10T00:56:00.779 INFO:teuthology.orchestra.run.smithi106.stderr: 88.2% -- replaced with /var/log/ceph/75c37498-de77-11ee-95c7-87774f69a715/ceph-mgr.y.log.gz 2024-03-10T00:56:00.804 INFO:teuthology.orchestra.run.smithi170.stderr: 93.1% -- replaced with /var/log/ceph/75c37498-de77-11ee-95c7-87774f69a715/ceph-osd.5.log.gz 2024-03-10T00:56:00.816 INFO:teuthology.orchestra.run.smithi106.stderr: 92.3% -- replaced with /var/log/ceph/75c37498-de77-11ee-95c7-87774f69a715/ceph-mon.c.log.gz 2024-03-10T00:56:00.831 INFO:teuthology.orchestra.run.smithi170.stderr: 93.0% -- replaced with /var/log/ceph/75c37498-de77-11ee-95c7-87774f69a715/ceph-osd.6.log.gz 2024-03-10T00:56:00.835 INFO:teuthology.orchestra.run.smithi170.stderr: 93.2% -- replaced with /var/log/ceph/75c37498-de77-11ee-95c7-87774f69a715/ceph-osd.7.log.gz 2024-03-10T00:56:00.854 INFO:teuthology.orchestra.run.smithi106.stderr: 93.0% -- replaced with /var/log/ceph/75c37498-de77-11ee-95c7-87774f69a715/ceph-osd.2.log.gz 2024-03-10T00:56:00.892 INFO:teuthology.orchestra.run.smithi170.stderr: 93.5% -- replaced with /var/log/ceph/75c37498-de77-11ee-95c7-87774f69a715/ceph-osd.4.log.gz 2024-03-10T00:56:00.893 INFO:teuthology.orchestra.run.smithi170.stderr: 2024-03-10T00:56:00.894 INFO:teuthology.orchestra.run.smithi170.stderr:real 0m0.286s 2024-03-10T00:56:00.894 INFO:teuthology.orchestra.run.smithi170.stderr:user 0m1.017s 2024-03-10T00:56:00.894 INFO:teuthology.orchestra.run.smithi170.stderr:sys 0m0.060s 2024-03-10T00:56:00.897 INFO:teuthology.orchestra.run.smithi106.stderr: 93.4% -- replaced with /var/log/ceph/75c37498-de77-11ee-95c7-87774f69a715/ceph-osd.0.log.gz 2024-03-10T00:56:00.956 INFO:teuthology.orchestra.run.smithi106.stderr: 93.5% -- replaced with /var/log/ceph/75c37498-de77-11ee-95c7-87774f69a715/ceph-osd.1.log.gz 2024-03-10T00:56:00.986 INFO:teuthology.orchestra.run.smithi106.stderr: 93.5% -- replaced with /var/log/ceph/75c37498-de77-11ee-95c7-87774f69a715/ceph-osd.3.log.gz 2024-03-10T00:56:01.048 INFO:teuthology.orchestra.run.smithi106.stderr: 91.2% -- replaced with /var/log/ceph/75c37498-de77-11ee-95c7-87774f69a715/ceph-mon.a.log.gz 2024-03-10T00:56:01.050 INFO:teuthology.orchestra.run.smithi106.stderr: 2024-03-10T00:56:01.050 INFO:teuthology.orchestra.run.smithi106.stderr:real 0m0.445s 2024-03-10T00:56:01.050 INFO:teuthology.orchestra.run.smithi106.stderr:user 0m1.919s 2024-03-10T00:56:01.050 INFO:teuthology.orchestra.run.smithi106.stderr:sys 0m0.089s 2024-03-10T00:56:01.051 INFO:tasks.cephadm:Archiving logs... 2024-03-10T00:56:01.051 DEBUG:teuthology.misc:Transferring archived files from smithi106:/var/log/ceph to /home/teuthworker/archive/yuriw-2024-03-08_16:20:46-rados-wip-yuri4-testing-2024-03-05-0854-distro-default-smithi/7587708/remote/smithi106/log 2024-03-10T00:56:01.052 DEBUG:teuthology.orchestra.run.smithi106:> sudo tar c -f - -C /var/log/ceph -- . 2024-03-10T00:56:01.390 DEBUG:teuthology.misc:Transferring archived files from smithi170:/var/log/ceph to /home/teuthworker/archive/yuriw-2024-03-08_16:20:46-rados-wip-yuri4-testing-2024-03-05-0854-distro-default-smithi/7587708/remote/smithi170/log 2024-03-10T00:56:01.391 DEBUG:teuthology.orchestra.run.smithi170:> sudo tar c -f - -C /var/log/ceph -- . 2024-03-10T00:56:01.555 INFO:tasks.cephadm:Removing cluster... 2024-03-10T00:56:01.555 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 75c37498-de77-11ee-95c7-87774f69a715 --force 2024-03-10T00:56:01.731 INFO:teuthology.orchestra.run.smithi106.stdout:Deleting cluster with fsid: 75c37498-de77-11ee-95c7-87774f69a715 2024-03-10T00:56:01.854 DEBUG:teuthology.orchestra.run.smithi170:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 75c37498-de77-11ee-95c7-87774f69a715 --force 2024-03-10T00:56:02.033 INFO:teuthology.orchestra.run.smithi170.stdout:Deleting cluster with fsid: 75c37498-de77-11ee-95c7-87774f69a715 2024-03-10T00:56:02.156 INFO:tasks.cephadm:Removing cephadm ... 2024-03-10T00:56:02.156 DEBUG:teuthology.orchestra.run.smithi106:> rm -rf /home/ubuntu/cephtest/cephadm 2024-03-10T00:56:02.172 DEBUG:teuthology.orchestra.run.smithi170:> rm -rf /home/ubuntu/cephtest/cephadm 2024-03-10T00:56:02.187 INFO:tasks.cephadm:Teardown complete 2024-03-10T00:56:02.187 DEBUG:teuthology.run_tasks:Unwinding manager nvme_loop 2024-03-10T00:56:02.208 INFO:tasks.nvme_loop:Disconnecting nvme_loop smithi106:/dev/vg_nvme/lv_1... 2024-03-10T00:56:02.208 DEBUG:teuthology.orchestra.run.smithi106:> sudo nvme disconnect -n lv_1 2024-03-10T00:56:02.355 INFO:teuthology.orchestra.run.smithi106.stdout:NQN:lv_1 disconnected 1 controller(s) 2024-03-10T00:56:02.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-10T00:56:02.358 INFO:tasks.nvme_loop:Disconnecting nvme_loop smithi106:/dev/vg_nvme/lv_2... 2024-03-10T00:56:02.358 DEBUG:teuthology.orchestra.run.smithi106:> sudo nvme disconnect -n lv_2 2024-03-10T00:56:02.499 INFO:teuthology.orchestra.run.smithi106.stdout:NQN:lv_2 disconnected 1 controller(s) 2024-03-10T00:56:02.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-10T00:56:02.501 INFO:tasks.nvme_loop:Disconnecting nvme_loop smithi106:/dev/vg_nvme/lv_3... 2024-03-10T00:56:02.501 DEBUG:teuthology.orchestra.run.smithi106:> sudo nvme disconnect -n lv_3 2024-03-10T00:56:02.646 INFO:teuthology.orchestra.run.smithi106.stdout:NQN:lv_3 disconnected 1 controller(s) 2024-03-10T00:56:02.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-10T00:56:02.648 INFO:tasks.nvme_loop:Disconnecting nvme_loop smithi106:/dev/vg_nvme/lv_4... 2024-03-10T00:56:02.648 DEBUG:teuthology.orchestra.run.smithi106:> sudo nvme disconnect -n lv_4 2024-03-10T00:56:02.785 INFO:teuthology.orchestra.run.smithi106.stdout:NQN:lv_4 disconnected 1 controller(s) 2024-03-10T00:56:02.787 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-10T00:56:02.787 DEBUG:teuthology.orchestra.run.smithi106:> set -ex 2024-03-10T00:56:02.787 DEBUG:teuthology.orchestra.run.smithi106:> sudo dd of=/scratch_devs 2024-03-10T00:56:02.814 INFO:tasks.nvme_loop:Disconnecting nvme_loop smithi170:/dev/vg_nvme/lv_1... 2024-03-10T00:56:02.815 DEBUG:teuthology.orchestra.run.smithi170:> sudo nvme disconnect -n lv_1 2024-03-10T00:56:02.965 INFO:teuthology.orchestra.run.smithi170.stdout:NQN:lv_1 disconnected 1 controller(s) 2024-03-10T00:56:02.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-10T00:56:02.966 INFO:tasks.nvme_loop:Disconnecting nvme_loop smithi170:/dev/vg_nvme/lv_2... 2024-03-10T00:56:02.967 DEBUG:teuthology.orchestra.run.smithi170:> sudo nvme disconnect -n lv_2 2024-03-10T00:56:03.093 INFO:teuthology.orchestra.run.smithi170.stdout:NQN:lv_2 disconnected 1 controller(s) 2024-03-10T00:56:03.094 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-10T00:56:03.095 INFO:tasks.nvme_loop:Disconnecting nvme_loop smithi170:/dev/vg_nvme/lv_3... 2024-03-10T00:56:03.095 DEBUG:teuthology.orchestra.run.smithi170:> sudo nvme disconnect -n lv_3 2024-03-10T00:56:03.274 INFO:teuthology.orchestra.run.smithi170.stdout:NQN:lv_3 disconnected 1 controller(s) 2024-03-10T00:56:03.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-10T00:56:03.276 INFO:tasks.nvme_loop:Disconnecting nvme_loop smithi170:/dev/vg_nvme/lv_4... 2024-03-10T00:56:03.276 DEBUG:teuthology.orchestra.run.smithi170:> sudo nvme disconnect -n lv_4 2024-03-10T00:56:03.431 INFO:teuthology.orchestra.run.smithi170.stdout:NQN:lv_4 disconnected 1 controller(s) 2024-03-10T00:56:03.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-10T00:56:03.433 DEBUG:teuthology.orchestra.run.smithi170:> set -ex 2024-03-10T00:56:03.433 DEBUG:teuthology.orchestra.run.smithi170:> sudo dd of=/scratch_devs 2024-03-10T00:56:03.461 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-03-10T00:56:03.470 INFO:teuthology.task.clock:Checking final clock skew... 2024-03-10T00:56:03.471 DEBUG:teuthology.orchestra.run.smithi106:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-10T00:56:03.473 DEBUG:teuthology.orchestra.run.smithi170:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-10T00:56:03.486 INFO:teuthology.orchestra.run.smithi106.stderr:bash: line 1: ntpq: command not found 2024-03-10T00:56:03.490 INFO:teuthology.orchestra.run.smithi106.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-10T00:56:03.490 INFO:teuthology.orchestra.run.smithi106.stdout:=============================================================================== 2024-03-10T00:56:03.490 INFO:teuthology.orchestra.run.smithi106.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 0 +2443us[+2443us] +/- 117ms 2024-03-10T00:56:03.491 INFO:teuthology.orchestra.run.smithi106.stdout:^* hv02.front.sepia.ceph.com 3 6 377 2 -739us[ -736us] +/- 53ms 2024-03-10T00:56:03.491 INFO:teuthology.orchestra.run.smithi106.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 1 -477us[ -477us] +/- 87ms 2024-03-10T00:56:03.491 INFO:teuthology.orchestra.run.smithi106.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2024-03-10T00:56:03.515 INFO:teuthology.orchestra.run.smithi170.stderr:bash: line 1: ntpq: command not found 2024-03-10T00:56:03.520 INFO:teuthology.orchestra.run.smithi170.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-10T00:56:03.520 INFO:teuthology.orchestra.run.smithi170.stdout:=============================================================================== 2024-03-10T00:56:03.520 INFO:teuthology.orchestra.run.smithi170.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 2 +2414us[+2414us] +/- 117ms 2024-03-10T00:56:03.520 INFO:teuthology.orchestra.run.smithi170.stdout:^* hv02.front.sepia.ceph.com 3 6 377 2 -779us[ -775us] +/- 53ms 2024-03-10T00:56:03.520 INFO:teuthology.orchestra.run.smithi170.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 2 -502us[ -502us] +/- 87ms 2024-03-10T00:56:03.520 INFO:teuthology.orchestra.run.smithi170.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2024-03-10T00:56:03.520 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-03-10T00:56:03.529 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-03-10T00:56:03.530 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-03-10T00:56:03.571 DEBUG:teuthology.orchestra.run.smithi106:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-03-10T00:56:03.652 DEBUG:teuthology.orchestra.run.smithi170:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-03-10T00:56:03.731 DEBUG:teuthology.orchestra.run.smithi106:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-10T00:56:03.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-10T00:56:03.762 DEBUG:teuthology.orchestra.run.smithi170:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-10T00:56:03.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-10T00:56:03.793 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-03-10T00:56:03.802 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-03-10T00:56:03.810 INFO:teuthology.task.internal:Duration was 908.992811 seconds 2024-03-10T00:56:03.811 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-03-10T00:56:03.819 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-03-10T00:56:03.819 DEBUG:teuthology.orchestra.run.smithi106:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-03-10T00:56:03.821 DEBUG:teuthology.orchestra.run.smithi170:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-03-10T00:56:03.862 INFO:teuthology.orchestra.run.smithi106.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-10T00:56:03.876 INFO:teuthology.orchestra.run.smithi170.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-10T00:56:04.380 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-03-10T00:56:04.380 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi106.front.sepia.ceph.com 2024-03-10T00:56:04.381 DEBUG:teuthology.orchestra.run.smithi106:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-03-10T00:56:04.440 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi170.front.sepia.ceph.com 2024-03-10T00:56:04.440 DEBUG:teuthology.orchestra.run.smithi170:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-03-10T00:56:04.469 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-03-10T00:56:04.469 DEBUG:teuthology.orchestra.run.smithi106:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-03-10T00:56:04.482 DEBUG:teuthology.orchestra.run.smithi170:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-03-10T00:56:04.536 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-03-10T00:56:04.537 DEBUG:teuthology.orchestra.run.smithi106:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-03-10T00:56:04.552 DEBUG:teuthology.orchestra.run.smithi170:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-03-10T00:56:04.810 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-03-10T00:56:04.821 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-03-10T00:56:04.821 DEBUG:teuthology.orchestra.run.smithi106:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-03-10T00:56:04.847 DEBUG:teuthology.orchestra.run.smithi170:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-03-10T00:56:04.874 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-03-10T00:56:04.884 DEBUG:teuthology.orchestra.run.smithi106:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-03-10T00:56:04.889 DEBUG:teuthology.orchestra.run.smithi170:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-03-10T00:56:04.912 INFO:teuthology.orchestra.run.smithi106.stdout:kernel.core_pattern = core 2024-03-10T00:56:04.939 INFO:teuthology.orchestra.run.smithi170.stdout:kernel.core_pattern = core 2024-03-10T00:56:04.962 DEBUG:teuthology.orchestra.run.smithi106:> test -e /home/ubuntu/cephtest/archive/coredump 2024-03-10T00:56:05.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-10T00:56:05.003 DEBUG:teuthology.orchestra.run.smithi170:> test -e /home/ubuntu/cephtest/archive/coredump 2024-03-10T00:56:05.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-10T00:56:05.017 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-03-10T00:56:05.028 INFO:teuthology.task.internal:Transferring archived files... 2024-03-10T00:56:05.029 DEBUG:teuthology.misc:Transferring archived files from smithi106:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/yuriw-2024-03-08_16:20:46-rados-wip-yuri4-testing-2024-03-05-0854-distro-default-smithi/7587708/remote/smithi106 2024-03-10T00:56:05.029 DEBUG:teuthology.orchestra.run.smithi106:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-03-10T00:56:05.082 DEBUG:teuthology.misc:Transferring archived files from smithi170:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/yuriw-2024-03-08_16:20:46-rados-wip-yuri4-testing-2024-03-05-0854-distro-default-smithi/7587708/remote/smithi170 2024-03-10T00:56:05.083 DEBUG:teuthology.orchestra.run.smithi170:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-03-10T00:56:05.121 INFO:teuthology.task.internal:Removing archive directory... 2024-03-10T00:56:05.121 DEBUG:teuthology.orchestra.run.smithi106:> rm -rf -- /home/ubuntu/cephtest/archive 2024-03-10T00:56:05.124 DEBUG:teuthology.orchestra.run.smithi170:> rm -rf -- /home/ubuntu/cephtest/archive 2024-03-10T00:56:05.171 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-03-10T00:56:05.183 INFO:teuthology.task.internal:Not uploading archives. 2024-03-10T00:56:05.183 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-03-10T00:56:05.193 INFO:teuthology.task.internal:Tidying up after the test... 2024-03-10T00:56:05.193 DEBUG:teuthology.orchestra.run.smithi106:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-03-10T00:56:05.195 DEBUG:teuthology.orchestra.run.smithi170:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-03-10T00:56:05.207 INFO:teuthology.orchestra.run.smithi106.stdout: 265701 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 10 00:56 /home/ubuntu/cephtest 2024-03-10T00:56:05.225 INFO:teuthology.orchestra.run.smithi170.stdout: 262166 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 10 00:56 /home/ubuntu/cephtest 2024-03-10T00:56:05.227 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-03-10T00:56:05.238 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-03-10T00:56:05.258 INFO:teuthology.nuke:Checking targets against current locks 2024-03-10T00:56:05.287 DEBUG:teuthology.nuke:shortname: smithi106 2024-03-10T00:56:05.287 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-10T00:56:05.313 DEBUG:teuthology.nuke:shortname: smithi170 2024-03-10T00:56:05.313 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-10T00:56:05.316 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi106.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2024-03-08_16:20:46-rados-wip-yuri4-testing-2024-03-05-0854-distro-default-smithi/7587708', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-10 00:32:12.960846', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBET97ZX/gb8FSJnFNCUlMEvsQi2MTZvHo2itkZIAxRVC2oVnUb1UXZEjIzujvNU7ZF7j6gytQtN6TYgpvmNbdgQ='} 2024-03-10T00:56:05.339 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi170.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2024-03-08_16:20:46-rados-wip-yuri4-testing-2024-03-05-0854-distro-default-smithi/7587708', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-10 00:32:12.958913', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBGf5wFffxIrQcRI0K0bQr3WXg0F9WC1OE706cGuypPKp03l1oIpUATODTmpzKJAzWPPUXVcdGCWKDUvu+K+5h8o='} 2024-03-10T00:56:05.365 INFO:teuthology.orchestra.console.smithi106:Power off 2024-03-10T00:56:05.365 DEBUG:teuthology.orchestra.console.smithi106:pexpect command: ipmitool -H smithi106.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-10T00:56:05.391 DEBUG:teuthology.orchestra.console.smithi106:power off output: Chassis Power Control: Down/Off 2024-03-10T00:56:05.391 DEBUG:teuthology.orchestra.console.smithi106:pexpect command: ipmitool -H smithi106.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-10T00:56:05.409 INFO:teuthology.orchestra.console.smithi170:Power off 2024-03-10T00:56:05.409 DEBUG:teuthology.orchestra.console.smithi170:pexpect command: ipmitool -H smithi170.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-10T00:56:05.418 DEBUG:teuthology.orchestra.console.smithi106:check power output: Chassis Power is on 2024-03-10T00:56:05.433 DEBUG:teuthology.orchestra.console.smithi170:power off output: Chassis Power Control: Down/Off 2024-03-10T00:56:05.433 DEBUG:teuthology.orchestra.console.smithi170:pexpect command: ipmitool -H smithi170.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-10T00:56:05.455 DEBUG:teuthology.orchestra.console.smithi170:check power output: Chassis Power is on 2024-03-10T00:56:09.419 DEBUG:teuthology.orchestra.console.smithi106:pexpect command: ipmitool -H smithi106.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-10T00:56:09.457 DEBUG:teuthology.orchestra.console.smithi170:pexpect command: ipmitool -H smithi170.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-10T00:56:09.530 DEBUG:teuthology.orchestra.console.smithi106:check power output: Chassis Power is on 2024-03-10T00:56:09.567 DEBUG:teuthology.orchestra.console.smithi170:check power output: Chassis Power is on 2024-03-10T00:56:13.534 DEBUG:teuthology.orchestra.console.smithi106:pexpect command: ipmitool -H smithi106.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-10T00:56:13.568 DEBUG:teuthology.orchestra.console.smithi170:pexpect command: ipmitool -H smithi170.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-10T00:56:13.645 DEBUG:teuthology.orchestra.console.smithi106:check power output: Chassis Power is off 2024-03-10T00:56:13.679 DEBUG:teuthology.orchestra.console.smithi170:check power output: Chassis Power is off 2024-03-10T00:56:13.746 INFO:teuthology.orchestra.console.smithi106:Power off completed 2024-03-10T00:56:13.780 INFO:teuthology.orchestra.console.smithi170:Power off completed 2024-03-10T00:56:13.884 INFO:teuthology.run:Summary data: description: rados/cephadm/smoke/{0-distro/centos_9.stream_runc 0-nvme-loop agent/on fixed-2 mon_election/connectivity start} duration: 908.9928114414215 failure_reason: '"2024-03-10T00:50:47.468873+0000 mon.a (mon.0) 648 : cluster [WRN] Health check failed: 1 stray daemon(s) not managed by cephadm (CEPHADM_STRAY_DAEMON)" in cluster log' owner: scheduled_yuriw@teuthology success: false 2024-03-10T00:56:13.884 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-03-10T00:56:13.966 INFO:teuthology.run:FAIL