2024-03-20T08:30:58.093 INFO:root:teuthology version: 0.0.1.dev262+ge691533 2024-03-20T08:30:58.093 DEBUG:teuthology.run:Teuthology command: teuthology --verbose --archive /home/teuthworker/archive/yuriw-2024-03-19_00:09:45-rados-wip-yuri5-testing-2024-03-18-1144-distro-default-smithi/7610111 --owner scheduled_yuriw@teuthology --name yuriw-2024-03-19_00:09:45-rados-wip-yuri5-testing-2024-03-18-1144-distro-default-smithi --description rados/cephadm/smoke/{0-distro/centos_9.stream 0-nvme-loop agent/on fixed-2 mon_election/connectivity start} -- /home/teuthworker/archive/yuriw-2024-03-19_00:09:45-rados-wip-yuri5-testing-2024-03-18-1144-distro-default-smithi/7610111/orig.config.yaml 2024-03-20T08:30:58.121 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-03-20T08:30:58.200 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/yuriw-2024-03-19_00:09:45-rados-wip-yuri5-testing-2024-03-18-1144-distro-default-smithi/7610111 branch: wip-yuri5-testing-2024-03-18-1144 description: rados/cephadm/smoke/{0-distro/centos_9.stream 0-nvme-loop agent/on fixed-2 mon_election/connectivity start} email: yweinste@redhat.com first_in_suite: false job_id: '7610111' kernel: kdb: true sha1: distro last_in_suite: false machine_type: smithi name: yuriw-2024-03-19_00:09:45-rados-wip-yuri5-testing-2024-03-18-1144-distro-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 4 size: 10 os_type: centos os_version: 9.stream overrides: admin_socket: branch: wip-yuri5-testing-2024-03-18-1144 ceph: conf: global: mon election default strategy: 3 mgr: debug mgr: 20 debug ms: 1 mgr/cephadm/use_agent: true mon: debug mon: 20 debug ms: 1 debug paxos: 20 osd: debug ms: 1 debug osd: 20 osd shutdown pgref assert: true flavor: default log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - MON_DOWN sha1: e7e80a6ac9849f0f23432d0d9832f096730308ae ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} install: ceph: flavor: default sha1: e7e80a6ac9849f0f23432d0d9832f096730308ae selinux: allowlist: - scontext=system_u:system_r:logrotate_t:s0 workunit: branch: wip-yuri5-testing-2024-03-18-1144 sha1: e7e80a6ac9849f0f23432d0d9832f096730308ae owner: scheduled_yuriw@teuthology priority: 100 repo: https://github.com/ceph/ceph-ci.git roles: - - mon.a - mon.c - mgr.y - osd.0 - osd.1 - osd.2 - osd.3 - client.0 - ceph.rgw.foo.a - node-exporter.a - alertmanager.a - - mon.b - mgr.x - osd.4 - osd.5 - osd.6 - osd.7 - client.1 - prometheus.a - grafana.a - node-exporter.b - ceph.iscsi.iscsi.a seed: 7763 sha1: e7e80a6ac9849f0f23432d0d9832f096730308ae sleep_before_teardown: 0 subset: 111/120000 suite: rados suite_branch: wip-yuri5-testing-2024-03-18-1144 suite_path: /home/teuthworker/src/github.com_ceph_ceph-c_e7e80a6ac9849f0f23432d0d9832f096730308ae/qa suite_relpath: qa suite_repo: https://github.com/ceph/ceph-ci.git suite_sha1: e7e80a6ac9849f0f23432d0d9832f096730308ae targets: smithi183.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBE4gCTVxUfs527UFnLA/Kcux6hTHlOG8K4vYYK26DaPAdBmVfF4qKIk0niQNGp2Rmx7qYSVs5eD1tsF0kOzshpo= smithi194.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBLUGBjFRNsFwW1fGbSBzj8dKikSjLYtQonwzNpc8xquXi0ye2tjK6uCZ1gkUH4xO573o7ji0knEGNdy4ndlSL8g= tasks: - nvme_loop: null - cephadm: conf: mgr: debug mgr: 20 debug ms: 1 - cephadm.shell: mon.a: - stat -c '%u %g' /var/log/ceph | grep '167 167' - ceph orch status - ceph orch ps - ceph orch ls - ceph orch host ls - ceph orch device ls - ceph orch ls --format yaml - ceph orch ls | grep '^osd ' teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: e691533f9cbb33d85b2187bba20d7102f098636d timestamp: 2024-03-19_00:09:45 tube: smithi user: yuriw verbose: true worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 2024-03-20T08:30:58.200 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_ceph_ceph-c_e7e80a6ac9849f0f23432d0d9832f096730308ae/qa; will attempt to use it 2024-03-20T08:30:58.201 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_ceph_ceph-c_e7e80a6ac9849f0f23432d0d9832f096730308ae/qa/tasks 2024-03-20T08:30:58.201 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-03-20T08:30:58.203 INFO:teuthology.task.internal:Checking packages... 2024-03-20T08:30:58.226 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash 'e7e80a6ac9849f0f23432d0d9832f096730308ae' 2024-03-20T08:30:58.226 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-03-20T08:30:58.226 INFO:teuthology.packaging:ref: None 2024-03-20T08:30:58.226 INFO:teuthology.packaging:tag: None 2024-03-20T08:30:58.226 INFO:teuthology.packaging:branch: wip-yuri5-testing-2024-03-18-1144 2024-03-20T08:30:58.226 INFO:teuthology.packaging:sha1: e7e80a6ac9849f0f23432d0d9832f096730308ae 2024-03-20T08:30:58.227 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=wip-yuri5-testing-2024-03-18-1144 2024-03-20T08:30:58.365 INFO:teuthology.task.internal:Found packages for ceph version 19.0.0-2153.ge7e80a6a 2024-03-20T08:30:58.367 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-03-20T08:30:58.376 INFO:teuthology.task.internal:no buildpackages task found 2024-03-20T08:30:58.376 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-03-20T08:30:58.400 INFO:teuthology.task.internal:Saving configuration 2024-03-20T08:30:58.413 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-03-20T08:30:58.426 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-20T08:30:58.451 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi183.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2024-03-19_00:09:45-rados-wip-yuri5-testing-2024-03-18-1144-distro-default-smithi/7610111', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-20 08:26:00.972556', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBE4gCTVxUfs527UFnLA/Kcux6hTHlOG8K4vYYK26DaPAdBmVfF4qKIk0niQNGp2Rmx7qYSVs5eD1tsF0kOzshpo='} 2024-03-20T08:30:58.473 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi194.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2024-03-19_00:09:45-rados-wip-yuri5-testing-2024-03-18-1144-distro-default-smithi/7610111', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-20 08:26:00.974208', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBLUGBjFRNsFwW1fGbSBzj8dKikSjLYtQonwzNpc8xquXi0ye2tjK6uCZ1gkUH4xO573o7ji0knEGNdy4ndlSL8g='} 2024-03-20T08:30:58.473 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-03-20T08:30:58.480 INFO:teuthology.task.internal:roles: ubuntu@smithi183.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'ceph.rgw.foo.a', 'node-exporter.a', 'alertmanager.a'] 2024-03-20T08:30:58.480 INFO:teuthology.task.internal:roles: ubuntu@smithi194.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'grafana.a', 'node-exporter.b', 'ceph.iscsi.iscsi.a'] 2024-03-20T08:30:58.480 INFO:teuthology.run_tasks:Running task console_log... 2024-03-20T08:30:58.573 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f1c93290160>, signals=[15]) 2024-03-20T08:30:58.574 INFO:teuthology.run_tasks:Running task internal.connect... 2024-03-20T08:30:58.581 INFO:teuthology.task.internal:Opening connections... 2024-03-20T08:30:58.581 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi183.front.sepia.ceph.com 2024-03-20T08:30:58.583 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi183.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-20T08:30:58.684 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi194.front.sepia.ceph.com 2024-03-20T08:30:58.685 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi194.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-20T08:30:58.780 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-03-20T08:30:58.787 DEBUG:teuthology.orchestra.run.smithi183:> uname -m 2024-03-20T08:30:58.805 INFO:teuthology.orchestra.run.smithi183.stdout:x86_64 2024-03-20T08:30:58.805 DEBUG:teuthology.orchestra.run.smithi183:> cat /etc/os-release 2024-03-20T08:30:58.863 INFO:teuthology.orchestra.run.smithi183.stdout:NAME="CentOS Stream" 2024-03-20T08:30:58.863 INFO:teuthology.orchestra.run.smithi183.stdout:VERSION="9" 2024-03-20T08:30:58.863 INFO:teuthology.orchestra.run.smithi183.stdout:ID="centos" 2024-03-20T08:30:58.863 INFO:teuthology.orchestra.run.smithi183.stdout:ID_LIKE="rhel fedora" 2024-03-20T08:30:58.863 INFO:teuthology.orchestra.run.smithi183.stdout:VERSION_ID="9" 2024-03-20T08:30:58.863 INFO:teuthology.orchestra.run.smithi183.stdout:PLATFORM_ID="platform:el9" 2024-03-20T08:30:58.863 INFO:teuthology.orchestra.run.smithi183.stdout:PRETTY_NAME="CentOS Stream 9" 2024-03-20T08:30:58.863 INFO:teuthology.orchestra.run.smithi183.stdout:ANSI_COLOR="0;31" 2024-03-20T08:30:58.863 INFO:teuthology.orchestra.run.smithi183.stdout:LOGO="fedora-logo-icon" 2024-03-20T08:30:58.863 INFO:teuthology.orchestra.run.smithi183.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-03-20T08:30:58.863 INFO:teuthology.orchestra.run.smithi183.stdout:HOME_URL="https://centos.org/" 2024-03-20T08:30:58.864 INFO:teuthology.orchestra.run.smithi183.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-03-20T08:30:58.864 INFO:teuthology.orchestra.run.smithi183.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-03-20T08:30:58.864 INFO:teuthology.orchestra.run.smithi183.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-03-20T08:30:58.865 INFO:teuthology.lock.ops:Updating smithi183.front.sepia.ceph.com on lock server 2024-03-20T08:30:58.888 DEBUG:teuthology.orchestra.run.smithi194:> uname -m 2024-03-20T08:30:58.905 INFO:teuthology.orchestra.run.smithi194.stdout:x86_64 2024-03-20T08:30:58.905 DEBUG:teuthology.orchestra.run.smithi194:> cat /etc/os-release 2024-03-20T08:30:58.962 INFO:teuthology.orchestra.run.smithi194.stdout:NAME="CentOS Stream" 2024-03-20T08:30:58.963 INFO:teuthology.orchestra.run.smithi194.stdout:VERSION="9" 2024-03-20T08:30:58.963 INFO:teuthology.orchestra.run.smithi194.stdout:ID="centos" 2024-03-20T08:30:58.963 INFO:teuthology.orchestra.run.smithi194.stdout:ID_LIKE="rhel fedora" 2024-03-20T08:30:58.963 INFO:teuthology.orchestra.run.smithi194.stdout:VERSION_ID="9" 2024-03-20T08:30:58.964 INFO:teuthology.orchestra.run.smithi194.stdout:PLATFORM_ID="platform:el9" 2024-03-20T08:30:58.964 INFO:teuthology.orchestra.run.smithi194.stdout:PRETTY_NAME="CentOS Stream 9" 2024-03-20T08:30:58.964 INFO:teuthology.orchestra.run.smithi194.stdout:ANSI_COLOR="0;31" 2024-03-20T08:30:58.964 INFO:teuthology.orchestra.run.smithi194.stdout:LOGO="fedora-logo-icon" 2024-03-20T08:30:58.964 INFO:teuthology.orchestra.run.smithi194.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-03-20T08:30:58.964 INFO:teuthology.orchestra.run.smithi194.stdout:HOME_URL="https://centos.org/" 2024-03-20T08:30:58.964 INFO:teuthology.orchestra.run.smithi194.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-03-20T08:30:58.964 INFO:teuthology.orchestra.run.smithi194.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-03-20T08:30:58.964 INFO:teuthology.orchestra.run.smithi194.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-03-20T08:30:58.964 INFO:teuthology.lock.ops:Updating smithi194.front.sepia.ceph.com on lock server 2024-03-20T08:30:58.988 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-03-20T08:30:58.998 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-03-20T08:30:59.017 INFO:teuthology.task.internal:Checking for old test directory... 2024-03-20T08:30:59.017 DEBUG:teuthology.orchestra.run.smithi183:> test '!' -e /home/ubuntu/cephtest 2024-03-20T08:30:59.020 DEBUG:teuthology.orchestra.run.smithi194:> test '!' -e /home/ubuntu/cephtest 2024-03-20T08:30:59.036 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-03-20T08:30:59.042 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-03-20T08:30:59.042 DEBUG:teuthology.orchestra.run.smithi183:> test -z $(ls -A /var/lib/ceph) 2024-03-20T08:30:59.077 DEBUG:teuthology.orchestra.run.smithi194:> test -z $(ls -A /var/lib/ceph) 2024-03-20T08:30:59.104 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-03-20T08:30:59.161 INFO:teuthology.run_tasks:Running task kernel... 2024-03-20T08:30:59.176 INFO:teuthology.task.kernel:normalize config orig: {'kdb': True, 'sha1': 'distro'} 2024-03-20T08:30:59.176 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': True, 'sha1': 'distro'}, 'mon.b': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2024-03-20T08:30:59.176 DEBUG:teuthology.orchestra.run.smithi183:> test -f /run/.containerenv -o -f /.dockerenv 2024-03-20T08:30:59.177 DEBUG:teuthology.orchestra.run.smithi194:> test -f /run/.containerenv -o -f /.dockerenv 2024-03-20T08:30:59.192 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-20T08:30:59.193 DEBUG:teuthology.orchestra.run.smithi194:> uname -r 2024-03-20T08:30:59.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-20T08:30:59.194 DEBUG:teuthology.orchestra.run.smithi183:> uname -r 2024-03-20T08:30:59.252 INFO:teuthology.orchestra.run.smithi183.stdout:5.14.0-361.el9.x86_64 2024-03-20T08:30:59.252 INFO:teuthology.task.kernel:Running kernel on smithi183: 5.14.0-361.el9.x86_64 2024-03-20T08:30:59.252 DEBUG:teuthology.orchestra.run.smithi183:> sudo yum install -y kernel 2024-03-20T08:30:59.253 INFO:teuthology.orchestra.run.smithi194.stdout:5.14.0-361.el9.x86_64 2024-03-20T08:30:59.254 INFO:teuthology.task.kernel:Running kernel on smithi194: 5.14.0-361.el9.x86_64 2024-03-20T08:30:59.254 DEBUG:teuthology.orchestra.run.smithi194:> sudo yum install -y kernel 2024-03-20T08:31:01.322 INFO:teuthology.orchestra.run.smithi183.stdout:CentOS-9 - AppStream 14 MB/s | 6.7 MB 00:00 2024-03-20T08:31:01.371 INFO:teuthology.orchestra.run.smithi194.stdout:CentOS-9 - AppStream 13 MB/s | 6.7 MB 00:00 2024-03-20T08:31:02.964 INFO:teuthology.orchestra.run.smithi194.stdout:CentOS-9 - BaseOS 4.2 MB/s | 2.0 MB 00:00 2024-03-20T08:31:03.114 INFO:teuthology.orchestra.run.smithi183.stdout:CentOS-9 - BaseOS 2.9 MB/s | 2.0 MB 00:00 2024-03-20T08:31:03.929 INFO:teuthology.orchestra.run.smithi194.stdout:CentOS Stream 9 - BaseOS 13 MB/s | 8.0 MB 00:00 2024-03-20T08:31:05.821 INFO:teuthology.orchestra.run.smithi183.stdout:CentOS Stream 9 - BaseOS 3.4 MB/s | 8.0 MB 00:02 2024-03-20T08:31:06.064 INFO:teuthology.orchestra.run.smithi194.stdout:CentOS Stream 9 - AppStream 17 MB/s | 19 MB 00:01 2024-03-20T08:31:09.072 INFO:teuthology.orchestra.run.smithi183.stdout:CentOS Stream 9 - AppStream 8.5 MB/s | 19 MB 00:02 2024-03-20T08:31:10.731 INFO:teuthology.orchestra.run.smithi194.stdout:CentOS Stream 9 - CRB 5.0 MB/s | 6.1 MB 00:01 2024-03-20T08:31:12.307 INFO:teuthology.orchestra.run.smithi194.stdout:CentOS Stream 9 - Extras packages 36 kB/s | 16 kB 00:00 2024-03-20T08:31:13.089 INFO:teuthology.orchestra.run.smithi183.stdout:CentOS Stream 9 - CRB 10 MB/s | 6.1 MB 00:00 2024-03-20T08:31:14.565 INFO:teuthology.orchestra.run.smithi183.stdout:CentOS Stream 9 - Extras packages 45 kB/s | 16 kB 00:00 2024-03-20T08:31:15.610 INFO:teuthology.orchestra.run.smithi183.stdout:Extra Packages for Enterprise Linux 22 MB/s | 21 MB 00:00 2024-03-20T08:31:20.516 INFO:teuthology.orchestra.run.smithi183.stdout:lab-extras 12 kB/s | 1.7 kB 00:00 2024-03-20T08:31:22.622 INFO:teuthology.orchestra.run.smithi183.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-03-20T08:31:22.623 INFO:teuthology.orchestra.run.smithi183.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2024-03-20T08:31:22.717 INFO:teuthology.orchestra.run.smithi183.stdout:Dependencies resolved. 2024-03-20T08:31:22.727 INFO:teuthology.orchestra.run.smithi183.stdout:================================================================================ 2024-03-20T08:31:22.727 INFO:teuthology.orchestra.run.smithi183.stdout: Package Architecture Version Repository Size 2024-03-20T08:31:22.727 INFO:teuthology.orchestra.run.smithi183.stdout:================================================================================ 2024-03-20T08:31:22.727 INFO:teuthology.orchestra.run.smithi183.stdout:Installing: 2024-03-20T08:31:22.728 INFO:teuthology.orchestra.run.smithi183.stdout: kernel x86_64 5.14.0-430.el9 baseos 44 k 2024-03-20T08:31:22.728 INFO:teuthology.orchestra.run.smithi183.stdout:Installing dependencies: 2024-03-20T08:31:22.728 INFO:teuthology.orchestra.run.smithi183.stdout: kernel-core x86_64 5.14.0-430.el9 baseos 15 M 2024-03-20T08:31:22.728 INFO:teuthology.orchestra.run.smithi183.stdout: kernel-modules x86_64 5.14.0-430.el9 baseos 33 M 2024-03-20T08:31:22.728 INFO:teuthology.orchestra.run.smithi183.stdout: kernel-modules-core x86_64 5.14.0-430.el9 baseos 27 M 2024-03-20T08:31:22.728 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-20T08:31:22.728 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction Summary 2024-03-20T08:31:22.728 INFO:teuthology.orchestra.run.smithi183.stdout:================================================================================ 2024-03-20T08:31:22.728 INFO:teuthology.orchestra.run.smithi183.stdout:Install 4 Packages 2024-03-20T08:31:22.728 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-20T08:31:22.729 INFO:teuthology.orchestra.run.smithi183.stdout:Total download size: 76 M 2024-03-20T08:31:22.729 INFO:teuthology.orchestra.run.smithi183.stdout:Installed size: 123 M 2024-03-20T08:31:22.729 INFO:teuthology.orchestra.run.smithi183.stdout:Downloading Packages: 2024-03-20T08:31:23.142 INFO:teuthology.orchestra.run.smithi183.stdout:(1/4): kernel-5.14.0-430.el9.x86_64.rpm 370 kB/s | 44 kB 00:00 2024-03-20T08:31:25.475 INFO:teuthology.orchestra.run.smithi183.stdout:(2/4): kernel-core-5.14.0-430.el9.x86_64.rpm 6.1 MB/s | 15 MB 00:02 2024-03-20T08:31:26.834 INFO:teuthology.orchestra.run.smithi183.stdout:(3/4): kernel-modules-core-5.14.0-430.el9.x86_6 7.4 MB/s | 27 MB 00:03 2024-03-20T08:31:28.584 INFO:teuthology.orchestra.run.smithi183.stdout:(4/4): kernel-modules-5.14.0-430.el9.x86_64.rpm 6.0 MB/s | 33 MB 00:05 2024-03-20T08:31:28.584 INFO:teuthology.orchestra.run.smithi183.stdout:-------------------------------------------------------------------------------- 2024-03-20T08:31:28.584 INFO:teuthology.orchestra.run.smithi183.stdout:Total 13 MB/s | 76 MB 00:05 2024-03-20T08:31:29.252 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction check 2024-03-20T08:31:29.617 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction check succeeded. 2024-03-20T08:31:29.617 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction test 2024-03-20T08:31:30.651 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction test succeeded. 2024-03-20T08:31:30.652 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction 2024-03-20T08:31:32.841 INFO:teuthology.orchestra.run.smithi194.stdout:Extra Packages for Enterprise Linux 1.0 MB/s | 21 MB 00:20 2024-03-20T08:31:33.137 INFO:teuthology.orchestra.run.smithi183.stdout: Preparing : 1/1 2024-03-20T08:31:33.749 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : kernel-modules-core-5.14.0-430.el9.x86_64 1/4 2024-03-20T08:31:33.833 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : kernel-core-5.14.0-430.el9.x86_64 2/4 2024-03-20T08:31:36.628 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: kernel-core-5.14.0-430.el9.x86_64 2/4 2024-03-20T08:31:36.769 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : kernel-modules-5.14.0-430.el9.x86_64 3/4 2024-03-20T08:31:37.980 INFO:teuthology.orchestra.run.smithi194.stdout:lab-extras 14 kB/s | 1.7 kB 00:00 2024-03-20T08:31:40.053 INFO:teuthology.orchestra.run.smithi194.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-03-20T08:31:40.054 INFO:teuthology.orchestra.run.smithi194.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2024-03-20T08:31:40.153 INFO:teuthology.orchestra.run.smithi194.stdout:Dependencies resolved. 2024-03-20T08:31:40.158 INFO:teuthology.orchestra.run.smithi194.stdout:================================================================================ 2024-03-20T08:31:40.158 INFO:teuthology.orchestra.run.smithi194.stdout: Package Architecture Version Repository Size 2024-03-20T08:31:40.158 INFO:teuthology.orchestra.run.smithi194.stdout:================================================================================ 2024-03-20T08:31:40.158 INFO:teuthology.orchestra.run.smithi194.stdout:Installing: 2024-03-20T08:31:40.158 INFO:teuthology.orchestra.run.smithi194.stdout: kernel x86_64 5.14.0-430.el9 baseos 44 k 2024-03-20T08:31:40.158 INFO:teuthology.orchestra.run.smithi194.stdout:Installing dependencies: 2024-03-20T08:31:40.158 INFO:teuthology.orchestra.run.smithi194.stdout: kernel-core x86_64 5.14.0-430.el9 baseos 15 M 2024-03-20T08:31:40.158 INFO:teuthology.orchestra.run.smithi194.stdout: kernel-modules x86_64 5.14.0-430.el9 baseos 33 M 2024-03-20T08:31:40.158 INFO:teuthology.orchestra.run.smithi194.stdout: kernel-modules-core x86_64 5.14.0-430.el9 baseos 27 M 2024-03-20T08:31:40.158 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-03-20T08:31:40.159 INFO:teuthology.orchestra.run.smithi194.stdout:Transaction Summary 2024-03-20T08:31:40.159 INFO:teuthology.orchestra.run.smithi194.stdout:================================================================================ 2024-03-20T08:31:40.159 INFO:teuthology.orchestra.run.smithi194.stdout:Install 4 Packages 2024-03-20T08:31:40.159 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-03-20T08:31:40.159 INFO:teuthology.orchestra.run.smithi194.stdout:Total download size: 76 M 2024-03-20T08:31:40.159 INFO:teuthology.orchestra.run.smithi194.stdout:Installed size: 123 M 2024-03-20T08:31:40.159 INFO:teuthology.orchestra.run.smithi194.stdout:Downloading Packages: 2024-03-20T08:31:40.415 INFO:teuthology.orchestra.run.smithi194.stdout:(1/4): kernel-5.14.0-430.el9.x86_64.rpm 465 kB/s | 44 kB 00:00 2024-03-20T08:31:40.773 INFO:teuthology.orchestra.run.smithi194.stdout:(2/4): kernel-core-5.14.0-430.el9.x86_64.rpm 33 MB/s | 15 MB 00:00 2024-03-20T08:31:41.173 INFO:teuthology.orchestra.run.smithi194.stdout:(3/4): kernel-modules-core-5.14.0-430.el9.x86_6 36 MB/s | 27 MB 00:00 2024-03-20T08:31:41.454 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: kernel-modules-5.14.0-430.el9.x86_64 3/4 2024-03-20T08:31:41.473 INFO:teuthology.orchestra.run.smithi194.stdout:(4/4): kernel-modules-5.14.0-430.el9.x86_64.rpm 29 MB/s | 33 MB 00:01 2024-03-20T08:31:41.474 INFO:teuthology.orchestra.run.smithi194.stdout:-------------------------------------------------------------------------------- 2024-03-20T08:31:41.474 INFO:teuthology.orchestra.run.smithi194.stdout:Total 58 MB/s | 76 MB 00:01 2024-03-20T08:31:41.502 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : kernel-5.14.0-430.el9.x86_64 4/4 2024-03-20T08:31:42.139 INFO:teuthology.orchestra.run.smithi194.stdout:Running transaction check 2024-03-20T08:31:42.501 INFO:teuthology.orchestra.run.smithi194.stdout:Transaction check succeeded. 2024-03-20T08:31:42.502 INFO:teuthology.orchestra.run.smithi194.stdout:Running transaction test 2024-03-20T08:31:43.536 INFO:teuthology.orchestra.run.smithi194.stdout:Transaction test succeeded. 2024-03-20T08:31:43.537 INFO:teuthology.orchestra.run.smithi194.stdout:Running transaction 2024-03-20T08:31:46.109 INFO:teuthology.orchestra.run.smithi194.stdout: Preparing : 1/1 2024-03-20T08:31:46.254 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: kernel-modules-core-5.14.0-430.el9.x86_64 4/4 2024-03-20T08:31:46.696 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : kernel-modules-core-5.14.0-430.el9.x86_64 1/4 2024-03-20T08:31:46.781 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : kernel-core-5.14.0-430.el9.x86_64 2/4 2024-03-20T08:31:49.500 INFO:teuthology.orchestra.run.smithi194.stdout: Running scriptlet: kernel-core-5.14.0-430.el9.x86_64 2/4 2024-03-20T08:31:49.642 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : kernel-modules-5.14.0-430.el9.x86_64 3/4 2024-03-20T08:31:54.334 INFO:teuthology.orchestra.run.smithi194.stdout: Running scriptlet: kernel-modules-5.14.0-430.el9.x86_64 3/4 2024-03-20T08:31:54.542 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : kernel-5.14.0-430.el9.x86_64 4/4 2024-03-20T08:31:59.298 INFO:teuthology.orchestra.run.smithi194.stdout: Running scriptlet: kernel-modules-core-5.14.0-430.el9.x86_64 4/4 2024-03-20T08:32:15.476 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: kernel-core-5.14.0-430.el9.x86_64 4/4 2024-03-20T08:32:15.476 INFO:teuthology.orchestra.run.smithi183.stdout:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-03-20T08:32:15.476 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-20T08:32:15.521 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: kernel-modules-5.14.0-430.el9.x86_64 4/4 2024-03-20T08:32:15.895 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: kernel-5.14.0-430.el9.x86_64 4/4 2024-03-20T08:32:15.895 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : kernel-5.14.0-430.el9.x86_64 1/4 2024-03-20T08:32:15.895 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : kernel-core-5.14.0-430.el9.x86_64 2/4 2024-03-20T08:32:15.895 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : kernel-modules-5.14.0-430.el9.x86_64 3/4 2024-03-20T08:32:16.294 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : kernel-modules-core-5.14.0-430.el9.x86_64 4/4 2024-03-20T08:32:16.294 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-20T08:32:16.294 INFO:teuthology.orchestra.run.smithi183.stdout:Installed: 2024-03-20T08:32:16.294 INFO:teuthology.orchestra.run.smithi183.stdout: kernel-5.14.0-430.el9.x86_64 2024-03-20T08:32:16.294 INFO:teuthology.orchestra.run.smithi183.stdout: kernel-core-5.14.0-430.el9.x86_64 2024-03-20T08:32:16.294 INFO:teuthology.orchestra.run.smithi183.stdout: kernel-modules-5.14.0-430.el9.x86_64 2024-03-20T08:32:16.294 INFO:teuthology.orchestra.run.smithi183.stdout: kernel-modules-core-5.14.0-430.el9.x86_64 2024-03-20T08:32:16.294 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-20T08:32:16.295 INFO:teuthology.orchestra.run.smithi183.stdout:Complete! 2024-03-20T08:32:16.637 DEBUG:teuthology.orchestra.run.smithi183:> rpm -q kernel | sort -rV | head -n 1 2024-03-20T08:32:16.733 INFO:teuthology.orchestra.run.smithi183.stdout:kernel-5.14.0-430.el9.x86_64 2024-03-20T08:32:16.733 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-430.el9.x86_64 2024-03-20T08:32:16.733 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-430.el9.x86_64 2024-03-20T08:32:16.734 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-03-20T08:32:16.734 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-03-20T08:32:16.734 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2024-03-20T08:32:16.734 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi183.front.sepia.ceph.com, path=None, version=distro) 2024-03-20T08:32:16.734 DEBUG:teuthology.orchestra.run.smithi183:> rpm -q kernel | sort -rV | head -n 1 2024-03-20T08:32:16.813 INFO:teuthology.orchestra.run.smithi183.stdout:kernel-5.14.0-430.el9.x86_64 2024-03-20T08:32:16.813 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-430.el9.x86_64 2024-03-20T08:32:16.813 DEBUG:teuthology.orchestra.run.smithi183:> sudo rpm -qi grub2-tools 2024-03-20T08:32:16.847 INFO:teuthology.orchestra.run.smithi183.stdout:Name : grub2-tools 2024-03-20T08:32:16.847 INFO:teuthology.orchestra.run.smithi183.stdout:Epoch : 1 2024-03-20T08:32:16.847 INFO:teuthology.orchestra.run.smithi183.stdout:Version : 2.06 2024-03-20T08:32:16.847 INFO:teuthology.orchestra.run.smithi183.stdout:Release : 68.el9 2024-03-20T08:32:16.847 INFO:teuthology.orchestra.run.smithi183.stdout:Architecture: x86_64 2024-03-20T08:32:16.847 INFO:teuthology.orchestra.run.smithi183.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-03-20T08:32:16.848 INFO:teuthology.orchestra.run.smithi183.stdout:Group : Unspecified 2024-03-20T08:32:16.848 INFO:teuthology.orchestra.run.smithi183.stdout:Size : 8264254 2024-03-20T08:32:16.848 INFO:teuthology.orchestra.run.smithi183.stdout:License : GPLv3+ 2024-03-20T08:32:16.848 INFO:teuthology.orchestra.run.smithi183.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-03-20T08:32:16.848 INFO:teuthology.orchestra.run.smithi183.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-03-20T08:32:16.848 INFO:teuthology.orchestra.run.smithi183.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-03-20T08:32:16.848 INFO:teuthology.orchestra.run.smithi183.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-03-20T08:32:16.848 INFO:teuthology.orchestra.run.smithi183.stdout:Packager : builder@centos.org 2024-03-20T08:32:16.848 INFO:teuthology.orchestra.run.smithi183.stdout:Vendor : CentOS 2024-03-20T08:32:16.848 INFO:teuthology.orchestra.run.smithi183.stdout:URL : http://www.gnu.org/software/grub/ 2024-03-20T08:32:16.848 INFO:teuthology.orchestra.run.smithi183.stdout:Summary : Support tools for GRUB. 2024-03-20T08:32:16.848 INFO:teuthology.orchestra.run.smithi183.stdout:Description : 2024-03-20T08:32:16.848 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-20T08:32:16.848 INFO:teuthology.orchestra.run.smithi183.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-03-20T08:32:16.848 INFO:teuthology.orchestra.run.smithi183.stdout:customizable bootloader with modular architecture. It supports a rich 2024-03-20T08:32:16.849 INFO:teuthology.orchestra.run.smithi183.stdout:variety of kernel formats, file systems, computer architectures and 2024-03-20T08:32:16.849 INFO:teuthology.orchestra.run.smithi183.stdout:hardware devices. 2024-03-20T08:32:16.849 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-20T08:32:16.849 INFO:teuthology.orchestra.run.smithi183.stdout:This subpackage provides tools for support of all platforms. 2024-03-20T08:32:16.849 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-03-20T08:32:16.850 INFO:teuthology.task.kernel:Updating grub on smithi183 to boot 5.14.0-430.el9.x86_64 2024-03-20T08:32:16.850 DEBUG:teuthology.orchestra.run.smithi183:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-03-20T08:32:17.433 INFO:teuthology.orchestra.run.smithi183.stderr:Generating grub configuration file ... 2024-03-20T08:32:18.784 INFO:teuthology.orchestra.run.smithi183.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-03-20T08:32:18.817 INFO:teuthology.orchestra.run.smithi183.stderr:done 2024-03-20T08:32:18.819 DEBUG:teuthology.orchestra.run.smithi183:> mktemp 2024-03-20T08:32:18.836 INFO:teuthology.orchestra.run.smithi183.stdout:/tmp/tmp.RWcTtDM6TF 2024-03-20T08:32:18.836 DEBUG:teuthology.orchestra.run.smithi183:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.RWcTtDM6TF 2024-03-20T08:32:18.905 DEBUG:teuthology.orchestra.run.smithi183:> sudo chmod 0666 /tmp/tmp.RWcTtDM6TF 2024-03-20T08:32:19.055 DEBUG:teuthology.orchestra.remote:smithi183:/tmp/tmp.RWcTtDM6TF is 6KB 2024-03-20T08:32:19.078 DEBUG:teuthology.orchestra.run.smithi183:> rm -fr /tmp/tmp.RWcTtDM6TF 2024-03-20T08:32:19.092 DEBUG:teuthology.orchestra.run.smithi183:> sudo /bin/ls /boot/loader/entries || true 2024-03-20T08:32:19.166 INFO:teuthology.orchestra.run.smithi183.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-03-20T08:32:19.166 INFO:teuthology.orchestra.run.smithi183.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-03-20T08:32:19.166 INFO:teuthology.orchestra.run.smithi183.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-03-20T08:32:19.167 INFO:teuthology.orchestra.run.smithi183.stdout:15949e3c88704300882482eddfd7baae-5.14.0-430.el9.x86_64.conf 2024-03-20T08:32:19.168 DEBUG:teuthology.orchestra.run.smithi183:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-430.el9.x86_64 2024-03-20T08:32:19.323 DEBUG:teuthology.orchestra.run.smithi183:> sudo shutdown -r now 2024-03-20T08:32:28.566 INFO:teuthology.orchestra.run.smithi194.stdout: Running scriptlet: kernel-core-5.14.0-430.el9.x86_64 4/4 2024-03-20T08:32:28.566 INFO:teuthology.orchestra.run.smithi194.stdout:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-03-20T08:32:28.566 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-03-20T08:32:28.610 INFO:teuthology.orchestra.run.smithi194.stdout: Running scriptlet: kernel-modules-5.14.0-430.el9.x86_64 4/4 2024-03-20T08:32:28.960 INFO:teuthology.orchestra.run.smithi194.stdout: Running scriptlet: kernel-5.14.0-430.el9.x86_64 4/4 2024-03-20T08:32:28.960 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : kernel-5.14.0-430.el9.x86_64 1/4 2024-03-20T08:32:28.960 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : kernel-core-5.14.0-430.el9.x86_64 2/4 2024-03-20T08:32:28.960 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : kernel-modules-5.14.0-430.el9.x86_64 3/4 2024-03-20T08:32:29.382 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : kernel-modules-core-5.14.0-430.el9.x86_64 4/4 2024-03-20T08:32:29.382 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-03-20T08:32:29.382 INFO:teuthology.orchestra.run.smithi194.stdout:Installed: 2024-03-20T08:32:29.382 INFO:teuthology.orchestra.run.smithi194.stdout: kernel-5.14.0-430.el9.x86_64 2024-03-20T08:32:29.382 INFO:teuthology.orchestra.run.smithi194.stdout: kernel-core-5.14.0-430.el9.x86_64 2024-03-20T08:32:29.382 INFO:teuthology.orchestra.run.smithi194.stdout: kernel-modules-5.14.0-430.el9.x86_64 2024-03-20T08:32:29.382 INFO:teuthology.orchestra.run.smithi194.stdout: kernel-modules-core-5.14.0-430.el9.x86_64 2024-03-20T08:32:29.382 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-03-20T08:32:29.383 INFO:teuthology.orchestra.run.smithi194.stdout:Complete! 2024-03-20T08:32:29.699 DEBUG:teuthology.orchestra.run.smithi194:> rpm -q kernel | sort -rV | head -n 1 2024-03-20T08:32:29.817 INFO:teuthology.orchestra.run.smithi194.stdout:kernel-5.14.0-430.el9.x86_64 2024-03-20T08:32:29.817 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-430.el9.x86_64 2024-03-20T08:32:29.817 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-430.el9.x86_64 2024-03-20T08:32:29.817 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-03-20T08:32:29.818 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-03-20T08:32:29.818 INFO:teuthology.task.kernel:Installing distro kernel on mon.b... 2024-03-20T08:32:29.818 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi194.front.sepia.ceph.com, path=None, version=distro) 2024-03-20T08:32:29.818 DEBUG:teuthology.orchestra.run.smithi194:> rpm -q kernel | sort -rV | head -n 1 2024-03-20T08:32:29.898 INFO:teuthology.orchestra.run.smithi194.stdout:kernel-5.14.0-430.el9.x86_64 2024-03-20T08:32:29.898 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-430.el9.x86_64 2024-03-20T08:32:29.898 DEBUG:teuthology.orchestra.run.smithi194:> sudo rpm -qi grub2-tools 2024-03-20T08:32:29.943 INFO:teuthology.orchestra.run.smithi194.stdout:Name : grub2-tools 2024-03-20T08:32:29.943 INFO:teuthology.orchestra.run.smithi194.stdout:Epoch : 1 2024-03-20T08:32:29.943 INFO:teuthology.orchestra.run.smithi194.stdout:Version : 2.06 2024-03-20T08:32:29.943 INFO:teuthology.orchestra.run.smithi194.stdout:Release : 68.el9 2024-03-20T08:32:29.943 INFO:teuthology.orchestra.run.smithi194.stdout:Architecture: x86_64 2024-03-20T08:32:29.943 INFO:teuthology.orchestra.run.smithi194.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-03-20T08:32:29.943 INFO:teuthology.orchestra.run.smithi194.stdout:Group : Unspecified 2024-03-20T08:32:29.943 INFO:teuthology.orchestra.run.smithi194.stdout:Size : 8264254 2024-03-20T08:32:29.944 INFO:teuthology.orchestra.run.smithi194.stdout:License : GPLv3+ 2024-03-20T08:32:29.944 INFO:teuthology.orchestra.run.smithi194.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-03-20T08:32:29.944 INFO:teuthology.orchestra.run.smithi194.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-03-20T08:32:29.944 INFO:teuthology.orchestra.run.smithi194.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-03-20T08:32:29.944 INFO:teuthology.orchestra.run.smithi194.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-03-20T08:32:29.944 INFO:teuthology.orchestra.run.smithi194.stdout:Packager : builder@centos.org 2024-03-20T08:32:29.944 INFO:teuthology.orchestra.run.smithi194.stdout:Vendor : CentOS 2024-03-20T08:32:29.944 INFO:teuthology.orchestra.run.smithi194.stdout:URL : http://www.gnu.org/software/grub/ 2024-03-20T08:32:29.944 INFO:teuthology.orchestra.run.smithi194.stdout:Summary : Support tools for GRUB. 2024-03-20T08:32:29.944 INFO:teuthology.orchestra.run.smithi194.stdout:Description : 2024-03-20T08:32:29.944 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-03-20T08:32:29.944 INFO:teuthology.orchestra.run.smithi194.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-03-20T08:32:29.944 INFO:teuthology.orchestra.run.smithi194.stdout:customizable bootloader with modular architecture. It supports a rich 2024-03-20T08:32:29.944 INFO:teuthology.orchestra.run.smithi194.stdout:variety of kernel formats, file systems, computer architectures and 2024-03-20T08:32:29.945 INFO:teuthology.orchestra.run.smithi194.stdout:hardware devices. 2024-03-20T08:32:29.945 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-03-20T08:32:29.945 INFO:teuthology.orchestra.run.smithi194.stdout:This subpackage provides tools for support of all platforms. 2024-03-20T08:32:29.946 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-03-20T08:32:29.946 INFO:teuthology.task.kernel:Updating grub on smithi194 to boot 5.14.0-430.el9.x86_64 2024-03-20T08:32:29.946 DEBUG:teuthology.orchestra.run.smithi194:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-03-20T08:32:30.602 INFO:teuthology.orchestra.run.smithi194.stderr:Generating grub configuration file ... 2024-03-20T08:32:31.926 INFO:teuthology.orchestra.run.smithi194.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-03-20T08:32:31.964 INFO:teuthology.orchestra.run.smithi194.stderr:done 2024-03-20T08:32:31.966 DEBUG:teuthology.orchestra.run.smithi194:> mktemp 2024-03-20T08:32:31.982 INFO:teuthology.orchestra.run.smithi194.stdout:/tmp/tmp.iVUZck8lqw 2024-03-20T08:32:31.982 DEBUG:teuthology.orchestra.run.smithi194:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.iVUZck8lqw 2024-03-20T08:32:32.051 DEBUG:teuthology.orchestra.run.smithi194:> sudo chmod 0666 /tmp/tmp.iVUZck8lqw 2024-03-20T08:32:32.193 DEBUG:teuthology.orchestra.remote:smithi194:/tmp/tmp.iVUZck8lqw is 6KB 2024-03-20T08:32:32.212 DEBUG:teuthology.orchestra.run.smithi194:> rm -fr /tmp/tmp.iVUZck8lqw 2024-03-20T08:32:32.240 DEBUG:teuthology.orchestra.run.smithi194:> sudo /bin/ls /boot/loader/entries || true 2024-03-20T08:32:32.291 INFO:teuthology.orchestra.run.smithi194.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-03-20T08:32:32.292 INFO:teuthology.orchestra.run.smithi194.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-03-20T08:32:32.292 INFO:teuthology.orchestra.run.smithi194.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-03-20T08:32:32.292 INFO:teuthology.orchestra.run.smithi194.stdout:15949e3c88704300882482eddfd7baae-5.14.0-430.el9.x86_64.conf 2024-03-20T08:32:32.293 DEBUG:teuthology.orchestra.run.smithi194:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-430.el9.x86_64 2024-03-20T08:32:32.472 DEBUG:teuthology.orchestra.run.smithi194:> sudo shutdown -r now 2024-03-20T08:32:49.342 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2024-03-20T08:32:49.343 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi183.front.sepia.ceph.com' 2024-03-20T08:32:49.345 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi183.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-20T08:33:02.486 INFO:teuthology.task.kernel:Checking client mon.b for new kernel version... 2024-03-20T08:33:02.487 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi194.front.sepia.ceph.com' 2024-03-20T08:33:02.487 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi194.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-20T08:33:07.875 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.183 2024-03-20T08:33:16.881 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi183.front.sepia.ceph.com' 2024-03-20T08:33:16.882 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi183.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-20T08:33:20.926 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.194 2024-03-20T08:33:29.936 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi194.front.sepia.ceph.com' 2024-03-20T08:33:29.937 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi194.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-20T08:33:32.994 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.194 2024-03-20T08:33:45.006 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi194.front.sepia.ceph.com' 2024-03-20T08:33:45.007 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi194.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-20T08:34:16.885 DEBUG:teuthology.orchestra.remote:timed out 2024-03-20T08:34:16.954 DEBUG:teuthology.orchestra.run.smithi194:> true 2024-03-20T08:34:17.357 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi194.front.sepia.ceph.com' 2024-03-20T08:34:17.357 INFO:teuthology.task.kernel:Checking kernel version of mon.b, want "5.14.0-430.el9.x86_64"... 2024-03-20T08:34:17.357 DEBUG:teuthology.orchestra.run.smithi194:> uname -r 2024-03-20T08:34:17.372 INFO:teuthology.orchestra.run.smithi194.stdout:5.14.0-430.el9.x86_64 2024-03-20T08:34:17.372 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-430.el9.x86_64 vs 5.14.0-430.el9.x86_64 2024-03-20T08:34:17.372 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-03-20T08:34:17.372 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-03-20T08:34:18.373 INFO:teuthology.task.kernel:Enabling kdb on mon.b... 2024-03-20T08:34:18.373 DEBUG:teuthology.orchestra.run.smithi194:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-03-20T08:34:18.455 INFO:teuthology.orchestra.run.smithi194.stdout:ttyS1 2024-03-20T08:34:18.483 DEBUG:teuthology.parallel:result is None 2024-03-20T08:34:28.886 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi183.front.sepia.ceph.com' 2024-03-20T08:34:28.887 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi183.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-20T08:34:29.251 DEBUG:teuthology.orchestra.run.smithi183:> true 2024-03-20T08:34:29.661 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi183.front.sepia.ceph.com' 2024-03-20T08:34:29.661 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "5.14.0-430.el9.x86_64"... 2024-03-20T08:34:29.662 DEBUG:teuthology.orchestra.run.smithi183:> uname -r 2024-03-20T08:34:29.677 INFO:teuthology.orchestra.run.smithi183.stdout:5.14.0-430.el9.x86_64 2024-03-20T08:34:29.678 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-430.el9.x86_64 vs 5.14.0-430.el9.x86_64 2024-03-20T08:34:29.678 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-03-20T08:34:29.678 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-03-20T08:34:30.678 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2024-03-20T08:34:30.679 DEBUG:teuthology.orchestra.run.smithi183:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-03-20T08:34:30.771 INFO:teuthology.orchestra.run.smithi183.stdout:ttyS1 2024-03-20T08:34:30.802 DEBUG:teuthology.parallel:result is None 2024-03-20T08:34:30.802 INFO:teuthology.run_tasks:Running task internal.base... 2024-03-20T08:34:30.809 INFO:teuthology.task.internal:Creating test directory... 2024-03-20T08:34:30.809 DEBUG:teuthology.orchestra.run.smithi183:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-03-20T08:34:30.814 DEBUG:teuthology.orchestra.run.smithi194:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-03-20T08:34:30.833 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-03-20T08:34:30.841 INFO:teuthology.run_tasks:Running task internal.archive... 2024-03-20T08:34:30.876 INFO:teuthology.task.internal:Creating archive directory... 2024-03-20T08:34:30.876 DEBUG:teuthology.orchestra.run.smithi183:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-03-20T08:34:30.880 DEBUG:teuthology.orchestra.run.smithi194:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-03-20T08:34:30.934 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-03-20T08:34:30.942 INFO:teuthology.task.internal:Enabling coredump saving... 2024-03-20T08:34:30.942 DEBUG:teuthology.orchestra.run.smithi183:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-03-20T08:34:30.946 DEBUG:teuthology.orchestra.run.smithi194:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-03-20T08:34:30.979 INFO:teuthology.orchestra.run.smithi183.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-20T08:34:30.984 INFO:teuthology.orchestra.run.smithi194.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-20T08:34:30.993 INFO:teuthology.orchestra.run.smithi183.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-20T08:34:30.998 INFO:teuthology.orchestra.run.smithi194.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-20T08:34:31.000 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-03-20T08:34:31.008 INFO:teuthology.task.internal:Configuring sudo... 2024-03-20T08:34:31.008 DEBUG:teuthology.orchestra.run.smithi183:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-03-20T08:34:31.039 DEBUG:teuthology.orchestra.run.smithi194:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-03-20T08:34:31.074 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-03-20T08:34:31.084 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-03-20T08:34:31.084 DEBUG:teuthology.orchestra.run.smithi183:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-03-20T08:34:31.109 DEBUG:teuthology.orchestra.run.smithi194:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-03-20T08:34:31.131 DEBUG:teuthology.orchestra.run.smithi183:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-20T08:34:31.207 DEBUG:teuthology.orchestra.run.smithi183:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-20T08:34:31.261 DEBUG:teuthology.orchestra.run.smithi183:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-20T08:34:31.346 DEBUG:teuthology.orchestra.run.smithi183:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-20T08:34:31.393 DEBUG:teuthology.orchestra.run.smithi183:> set -ex 2024-03-20T08:34:31.393 DEBUG:teuthology.orchestra.run.smithi183:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-03-20T08:34:31.465 DEBUG:teuthology.orchestra.run.smithi194:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-20T08:34:31.510 DEBUG:teuthology.orchestra.run.smithi194:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-20T08:34:31.563 DEBUG:teuthology.orchestra.run.smithi194:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-20T08:34:31.646 DEBUG:teuthology.orchestra.run.smithi194:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-20T08:34:31.693 DEBUG:teuthology.orchestra.run.smithi194:> set -ex 2024-03-20T08:34:31.694 DEBUG:teuthology.orchestra.run.smithi194:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-03-20T08:34:31.763 DEBUG:teuthology.orchestra.run.smithi183:> sudo service rsyslog restart 2024-03-20T08:34:31.766 DEBUG:teuthology.orchestra.run.smithi194:> sudo service rsyslog restart 2024-03-20T08:34:31.820 INFO:teuthology.orchestra.run.smithi183.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-20T08:34:31.850 INFO:teuthology.orchestra.run.smithi194.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-20T08:34:32.236 INFO:teuthology.run_tasks:Running task internal.timer... 2024-03-20T08:34:32.245 INFO:teuthology.task.internal:Starting timer... 2024-03-20T08:34:32.245 INFO:teuthology.run_tasks:Running task pcp... 2024-03-20T08:34:32.298 INFO:teuthology.run_tasks:Running task selinux... 2024-03-20T08:34:32.330 DEBUG:teuthology.task:Applying overrides for task selinux: {'allowlist': ['scontext=system_u:system_r:logrotate_t:s0']} 2024-03-20T08:34:32.331 DEBUG:teuthology.orchestra.run.smithi183:> sudo service auditd rotate 2024-03-20T08:34:32.379 INFO:teuthology.orchestra.run.smithi183.stdout:Rotating logs: 2024-03-20T08:34:32.381 DEBUG:teuthology.orchestra.run.smithi194:> sudo service auditd rotate 2024-03-20T08:34:32.437 INFO:teuthology.orchestra.run.smithi194.stdout:Rotating logs: 2024-03-20T08:34:32.439 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-03-20T08:34:32.439 DEBUG:teuthology.orchestra.run.smithi183:> /usr/sbin/getenforce 2024-03-20T08:34:32.463 INFO:teuthology.orchestra.run.smithi183.stdout:Permissive 2024-03-20T08:34:32.463 DEBUG:teuthology.orchestra.run.smithi194:> /usr/sbin/getenforce 2024-03-20T08:34:32.489 INFO:teuthology.orchestra.run.smithi194.stdout:Permissive 2024-03-20T08:34:32.489 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi183.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi194.front.sepia.ceph.com': 'permissive'} 2024-03-20T08:34:32.489 DEBUG:teuthology.orchestra.run.smithi183:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 2024-03-20T08:34:32.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-20T08:34:32.533 DEBUG:teuthology.orchestra.run.smithi194:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 2024-03-20T08:34:32.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-20T08:34:32.562 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-03-20T08:34:32.562 DEBUG:teuthology.orchestra.run.smithi183:> sudo /usr/sbin/setenforce permissive 2024-03-20T08:34:32.607 DEBUG:teuthology.orchestra.run.smithi194:> sudo /usr/sbin/setenforce permissive 2024-03-20T08:34:32.640 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-03-20T08:34:32.651 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-03-20T08:34:32.670 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-03-20T08:34:32.671 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi183.front.sepia.ceph.com,smithi194.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-03-20T08:38:55.976 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi183.front.sepia.ceph.com'), Remote(name='ubuntu@smithi194.front.sepia.ceph.com')] 2024-03-20T08:38:55.978 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi183.front.sepia.ceph.com' 2024-03-20T08:38:55.979 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi183.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-20T08:38:56.069 DEBUG:teuthology.orchestra.run.smithi183:> true 2024-03-20T08:38:56.145 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi183.front.sepia.ceph.com' 2024-03-20T08:38:56.145 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi194.front.sepia.ceph.com' 2024-03-20T08:38:56.146 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi194.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-20T08:38:56.231 DEBUG:teuthology.orchestra.run.smithi194:> true 2024-03-20T08:38:56.310 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi194.front.sepia.ceph.com' 2024-03-20T08:38:56.310 INFO:teuthology.run_tasks:Running task clock... 2024-03-20T08:38:56.323 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-03-20T08:38:56.324 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-03-20T08:38:56.324 DEBUG:teuthology.orchestra.run.smithi183:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-20T08:38:56.326 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-03-20T08:38:56.326 DEBUG:teuthology.orchestra.run.smithi194:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-20T08:38:56.355 INFO:teuthology.orchestra.run.smithi183.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-03-20T08:38:56.371 INFO:teuthology.orchestra.run.smithi183.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-03-20T08:38:56.380 INFO:teuthology.orchestra.run.smithi194.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-03-20T08:38:56.397 INFO:teuthology.orchestra.run.smithi194.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-03-20T08:38:56.400 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: ntpd: command not found 2024-03-20T08:38:56.413 INFO:teuthology.orchestra.run.smithi183.stdout:506 Cannot talk to daemon 2024-03-20T08:38:56.426 INFO:teuthology.orchestra.run.smithi194.stderr:sudo: ntpd: command not found 2024-03-20T08:38:56.430 INFO:teuthology.orchestra.run.smithi183.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-03-20T08:38:56.439 INFO:teuthology.orchestra.run.smithi194.stdout:506 Cannot talk to daemon 2024-03-20T08:38:56.446 INFO:teuthology.orchestra.run.smithi183.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-03-20T08:38:56.456 INFO:teuthology.orchestra.run.smithi194.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-03-20T08:38:56.472 INFO:teuthology.orchestra.run.smithi194.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-03-20T08:38:56.504 INFO:teuthology.orchestra.run.smithi183.stderr:bash: line 1: ntpq: command not found 2024-03-20T08:38:56.507 INFO:teuthology.orchestra.run.smithi183.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-20T08:38:56.507 INFO:teuthology.orchestra.run.smithi183.stdout:=============================================================================== 2024-03-20T08:38:56.507 INFO:teuthology.orchestra.run.smithi183.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-20T08:38:56.507 INFO:teuthology.orchestra.run.smithi183.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-20T08:38:56.507 INFO:teuthology.orchestra.run.smithi183.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-20T08:38:56.507 INFO:teuthology.orchestra.run.smithi183.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-20T08:38:56.523 INFO:teuthology.orchestra.run.smithi194.stderr:bash: line 1: ntpq: command not found 2024-03-20T08:38:56.526 INFO:teuthology.orchestra.run.smithi194.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-20T08:38:56.526 INFO:teuthology.orchestra.run.smithi194.stdout:=============================================================================== 2024-03-20T08:38:56.527 INFO:teuthology.orchestra.run.smithi194.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-20T08:38:56.527 INFO:teuthology.orchestra.run.smithi194.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-20T08:38:56.527 INFO:teuthology.orchestra.run.smithi194.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-20T08:38:56.527 INFO:teuthology.orchestra.run.smithi194.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-20T08:38:56.528 INFO:teuthology.run_tasks:Running task nvme_loop... 2024-03-20T08:38:56.537 INFO:tasks.nvme_loop:Setting up nvme_loop on scratch devices... 2024-03-20T08:38:56.537 DEBUG:teuthology.orchestra.run.smithi183:> set -ex 2024-03-20T08:38:56.538 DEBUG:teuthology.orchestra.run.smithi183:> dd if=/scratch_devs of=/dev/stdout 2024-03-20T08:38:56.563 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-03-20T08:38:56.563 DEBUG:teuthology.orchestra.run.smithi183:> stat /dev/vg_nvme/lv_1 2024-03-20T08:38:56.618 INFO:teuthology.orchestra.run.smithi183.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-03-20T08:38:56.618 INFO:teuthology.orchestra.run.smithi183.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-20T08:38:56.618 INFO:teuthology.orchestra.run.smithi183.stdout:Device: 5h/5d Inode: 666 Links: 1 2024-03-20T08:38:56.618 INFO:teuthology.orchestra.run.smithi183.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-20T08:38:56.618 INFO:teuthology.orchestra.run.smithi183.stdout:Context: system_u:object_r:device_t:s0 2024-03-20T08:38:56.618 INFO:teuthology.orchestra.run.smithi183.stdout:Access: 2024-03-20 08:38:54.988258811 +0000 2024-03-20T08:38:56.618 INFO:teuthology.orchestra.run.smithi183.stdout:Modify: 2024-03-20 08:38:25.059104588 +0000 2024-03-20T08:38:56.618 INFO:teuthology.orchestra.run.smithi183.stdout:Change: 2024-03-20 08:38:25.059104588 +0000 2024-03-20T08:38:56.618 INFO:teuthology.orchestra.run.smithi183.stdout: Birth: 2024-03-20 08:38:25.059104588 +0000 2024-03-20T08:38:56.618 DEBUG:teuthology.orchestra.run.smithi183:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-03-20T08:38:56.683 INFO:teuthology.orchestra.run.smithi183.stderr:1+0 records in 2024-03-20T08:38:56.683 INFO:teuthology.orchestra.run.smithi183.stderr:1+0 records out 2024-03-20T08:38:56.683 INFO:teuthology.orchestra.run.smithi183.stderr:512 bytes copied, 0.00017666 s, 2.9 MB/s 2024-03-20T08:38:56.684 DEBUG:teuthology.orchestra.run.smithi183:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-03-20T08:38:56.740 DEBUG:teuthology.orchestra.run.smithi183:> stat /dev/vg_nvme/lv_2 2024-03-20T08:38:56.795 INFO:teuthology.orchestra.run.smithi183.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-03-20T08:38:56.795 INFO:teuthology.orchestra.run.smithi183.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-20T08:38:56.795 INFO:teuthology.orchestra.run.smithi183.stdout:Device: 5h/5d Inode: 687 Links: 1 2024-03-20T08:38:56.795 INFO:teuthology.orchestra.run.smithi183.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-20T08:38:56.795 INFO:teuthology.orchestra.run.smithi183.stdout:Context: system_u:object_r:device_t:s0 2024-03-20T08:38:56.795 INFO:teuthology.orchestra.run.smithi183.stdout:Access: 2024-03-20 08:38:54.988258811 +0000 2024-03-20T08:38:56.795 INFO:teuthology.orchestra.run.smithi183.stdout:Modify: 2024-03-20 08:38:25.412094612 +0000 2024-03-20T08:38:56.795 INFO:teuthology.orchestra.run.smithi183.stdout:Change: 2024-03-20 08:38:25.412094612 +0000 2024-03-20T08:38:56.796 INFO:teuthology.orchestra.run.smithi183.stdout: Birth: 2024-03-20 08:38:25.412094612 +0000 2024-03-20T08:38:56.796 DEBUG:teuthology.orchestra.run.smithi183:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-03-20T08:38:56.861 INFO:teuthology.orchestra.run.smithi183.stderr:1+0 records in 2024-03-20T08:38:56.861 INFO:teuthology.orchestra.run.smithi183.stderr:1+0 records out 2024-03-20T08:38:56.861 INFO:teuthology.orchestra.run.smithi183.stderr:512 bytes copied, 0.000200116 s, 2.6 MB/s 2024-03-20T08:38:56.862 DEBUG:teuthology.orchestra.run.smithi183:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-03-20T08:38:56.918 DEBUG:teuthology.orchestra.run.smithi183:> stat /dev/vg_nvme/lv_3 2024-03-20T08:38:56.973 INFO:teuthology.orchestra.run.smithi183.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-03-20T08:38:56.973 INFO:teuthology.orchestra.run.smithi183.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-20T08:38:56.973 INFO:teuthology.orchestra.run.smithi183.stdout:Device: 5h/5d Inode: 703 Links: 1 2024-03-20T08:38:56.973 INFO:teuthology.orchestra.run.smithi183.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-20T08:38:56.973 INFO:teuthology.orchestra.run.smithi183.stdout:Context: system_u:object_r:device_t:s0 2024-03-20T08:38:56.973 INFO:teuthology.orchestra.run.smithi183.stdout:Access: 2024-03-20 08:38:54.988258811 +0000 2024-03-20T08:38:56.973 INFO:teuthology.orchestra.run.smithi183.stdout:Modify: 2024-03-20 08:38:25.766084609 +0000 2024-03-20T08:38:56.973 INFO:teuthology.orchestra.run.smithi183.stdout:Change: 2024-03-20 08:38:25.766084609 +0000 2024-03-20T08:38:56.973 INFO:teuthology.orchestra.run.smithi183.stdout: Birth: 2024-03-20 08:38:25.766084609 +0000 2024-03-20T08:38:56.974 DEBUG:teuthology.orchestra.run.smithi183:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-03-20T08:38:57.038 INFO:teuthology.orchestra.run.smithi183.stderr:1+0 records in 2024-03-20T08:38:57.039 INFO:teuthology.orchestra.run.smithi183.stderr:1+0 records out 2024-03-20T08:38:57.039 INFO:teuthology.orchestra.run.smithi183.stderr:512 bytes copied, 0.000209309 s, 2.4 MB/s 2024-03-20T08:38:57.040 DEBUG:teuthology.orchestra.run.smithi183:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-03-20T08:38:57.096 DEBUG:teuthology.orchestra.run.smithi183:> stat /dev/vg_nvme/lv_4 2024-03-20T08:38:57.154 INFO:teuthology.orchestra.run.smithi183.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-03-20T08:38:57.155 INFO:teuthology.orchestra.run.smithi183.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-20T08:38:57.155 INFO:teuthology.orchestra.run.smithi183.stdout:Device: 5h/5d Inode: 723 Links: 1 2024-03-20T08:38:57.155 INFO:teuthology.orchestra.run.smithi183.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-20T08:38:57.155 INFO:teuthology.orchestra.run.smithi183.stdout:Context: system_u:object_r:device_t:s0 2024-03-20T08:38:57.155 INFO:teuthology.orchestra.run.smithi183.stdout:Access: 2024-03-20 08:38:54.988258811 +0000 2024-03-20T08:38:57.155 INFO:teuthology.orchestra.run.smithi183.stdout:Modify: 2024-03-20 08:38:26.120074605 +0000 2024-03-20T08:38:57.155 INFO:teuthology.orchestra.run.smithi183.stdout:Change: 2024-03-20 08:38:26.120074605 +0000 2024-03-20T08:38:57.155 INFO:teuthology.orchestra.run.smithi183.stdout: Birth: 2024-03-20 08:38:26.120074605 +0000 2024-03-20T08:38:57.155 DEBUG:teuthology.orchestra.run.smithi183:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-03-20T08:38:57.219 INFO:teuthology.orchestra.run.smithi183.stderr:1+0 records in 2024-03-20T08:38:57.220 INFO:teuthology.orchestra.run.smithi183.stderr:1+0 records out 2024-03-20T08:38:57.220 INFO:teuthology.orchestra.run.smithi183.stderr:512 bytes copied, 0.000207104 s, 2.5 MB/s 2024-03-20T08:38:57.221 DEBUG:teuthology.orchestra.run.smithi183:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-03-20T08:38:57.276 DEBUG:teuthology.orchestra.run.smithi183:> grep '^nvme_loop' /proc/modules || sudo modprobe nvme_loop && sudo mkdir -p /sys/kernel/config/nvmet/hosts/hostnqn && sudo mkdir -p /sys/kernel/config/nvmet/ports/1 && echo loop | sudo tee /sys/kernel/config/nvmet/ports/1/addr_trtype 2024-03-20T08:38:57.417 INFO:teuthology.orchestra.run.smithi183.stdout:loop 2024-03-20T08:38:57.419 INFO:tasks.nvme_loop:Connecting nvme_loop smithi183:/dev/vg_nvme/lv_1... 2024-03-20T08:38:57.419 DEBUG:teuthology.orchestra.run.smithi183:> sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_1 && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_1/attr_allow_any_host && sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_1/namespaces/1 && echo -n /dev/vg_nvme/lv_1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_1/namespaces/1/device_path && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_1/namespaces/1/enable && sudo ln -s /sys/kernel/config/nvmet/subsystems/lv_1 /sys/kernel/config/nvmet/ports/1/subsystems/lv_1 && sudo nvme connect -t loop -n lv_1 -q hostnqn 2024-03-20T08:38:57.456 INFO:teuthology.orchestra.run.smithi183.stdout:1 2024-03-20T08:38:57.493 INFO:teuthology.orchestra.run.smithi183.stdout:/dev/vg_nvme/lv_11 2024-03-20T08:38:57.522 INFO:tasks.nvme_loop:Connecting nvme_loop smithi183:/dev/vg_nvme/lv_2... 2024-03-20T08:38:57.523 DEBUG:teuthology.orchestra.run.smithi183:> sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_2 && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_2/attr_allow_any_host && sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_2/namespaces/1 && echo -n /dev/vg_nvme/lv_2 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_2/namespaces/1/device_path && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_2/namespaces/1/enable && sudo ln -s /sys/kernel/config/nvmet/subsystems/lv_2 /sys/kernel/config/nvmet/ports/1/subsystems/lv_2 && sudo nvme connect -t loop -n lv_2 -q hostnqn 2024-03-20T08:38:57.600 INFO:teuthology.orchestra.run.smithi183.stdout:1 2024-03-20T08:38:57.638 INFO:teuthology.orchestra.run.smithi183.stdout:/dev/vg_nvme/lv_21 2024-03-20T08:38:57.665 INFO:tasks.nvme_loop:Connecting nvme_loop smithi183:/dev/vg_nvme/lv_3... 2024-03-20T08:38:57.665 DEBUG:teuthology.orchestra.run.smithi183:> sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_3 && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_3/attr_allow_any_host && sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_3/namespaces/1 && echo -n /dev/vg_nvme/lv_3 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_3/namespaces/1/device_path && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_3/namespaces/1/enable && sudo ln -s /sys/kernel/config/nvmet/subsystems/lv_3 /sys/kernel/config/nvmet/ports/1/subsystems/lv_3 && sudo nvme connect -t loop -n lv_3 -q hostnqn 2024-03-20T08:38:57.743 INFO:teuthology.orchestra.run.smithi183.stdout:1 2024-03-20T08:38:57.781 INFO:teuthology.orchestra.run.smithi183.stdout:/dev/vg_nvme/lv_31 2024-03-20T08:38:57.808 INFO:tasks.nvme_loop:Connecting nvme_loop smithi183:/dev/vg_nvme/lv_4... 2024-03-20T08:38:57.809 DEBUG:teuthology.orchestra.run.smithi183:> sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_4 && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_4/attr_allow_any_host && sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_4/namespaces/1 && echo -n /dev/vg_nvme/lv_4 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_4/namespaces/1/device_path && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_4/namespaces/1/enable && sudo ln -s /sys/kernel/config/nvmet/subsystems/lv_4 /sys/kernel/config/nvmet/ports/1/subsystems/lv_4 && sudo nvme connect -t loop -n lv_4 -q hostnqn 2024-03-20T08:38:57.893 INFO:teuthology.orchestra.run.smithi183.stdout:1 2024-03-20T08:38:57.930 INFO:teuthology.orchestra.run.smithi183.stdout:/dev/vg_nvme/lv_41 2024-03-20T08:38:57.958 DEBUG:teuthology.orchestra.run.smithi183:> set -ex 2024-03-20T08:38:57.958 DEBUG:teuthology.orchestra.run.smithi183:> dd if=/scratch_devs of=/dev/stdout 2024-03-20T08:38:58.011 DEBUG:teuthology.orchestra.run.smithi183:> sudo nvme list 2024-03-20T08:38:58.076 INFO:teuthology.orchestra.run.smithi183.stdout:Node SN Model Namespace Usage Format FW Rev 2024-03-20T08:38:58.077 INFO:teuthology.orchestra.run.smithi183.stdout:---------------- -------------------- ---------------------------------------- --------- -------------------------- ---------------- -------- 2024-03-20T08:38:58.077 INFO:teuthology.orchestra.run.smithi183.stdout:/dev/nvme0n1 PHFT620500QJ400BGN INTEL SSDPEDMD400G4 1 400.09 GB / 400.09 GB 512 B + 0 B 8DV101H0 2024-03-20T08:38:58.077 INFO:teuthology.orchestra.run.smithi183.stdout:/dev/nvme1n1 c6a13abe82e79d34211d Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 5.14.0-4 2024-03-20T08:38:58.077 INFO:teuthology.orchestra.run.smithi183.stdout:/dev/nvme2n1 2df36353822a1af07008 Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 5.14.0-4 2024-03-20T08:38:58.077 INFO:teuthology.orchestra.run.smithi183.stdout:/dev/nvme3n1 0558c7b77ddc7e9de393 Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 5.14.0-4 2024-03-20T08:38:58.077 INFO:teuthology.orchestra.run.smithi183.stdout:/dev/nvme4n1 0c043d4074e8718332b1 Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 5.14.0-4 2024-03-20T08:38:58.078 INFO:tasks.nvme_loop:new_devs ['/dev/nvme1n1', '/dev/nvme2n1', '/dev/nvme3n1', '/dev/nvme4n1'] 2024-03-20T08:38:58.078 DEBUG:teuthology.orchestra.run.smithi183:> set -ex 2024-03-20T08:38:58.078 DEBUG:teuthology.orchestra.run.smithi183:> sudo dd of=/scratch_devs 2024-03-20T08:38:58.145 DEBUG:teuthology.orchestra.run.smithi194:> set -ex 2024-03-20T08:38:58.145 DEBUG:teuthology.orchestra.run.smithi194:> dd if=/scratch_devs of=/dev/stdout 2024-03-20T08:38:58.163 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-03-20T08:38:58.164 DEBUG:teuthology.orchestra.run.smithi194:> stat /dev/vg_nvme/lv_1 2024-03-20T08:38:58.221 INFO:teuthology.orchestra.run.smithi194.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-03-20T08:38:58.221 INFO:teuthology.orchestra.run.smithi194.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-20T08:38:58.222 INFO:teuthology.orchestra.run.smithi194.stdout:Device: 5h/5d Inode: 667 Links: 1 2024-03-20T08:38:58.222 INFO:teuthology.orchestra.run.smithi194.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-20T08:38:58.222 INFO:teuthology.orchestra.run.smithi194.stdout:Context: system_u:object_r:device_t:s0 2024-03-20T08:38:58.222 INFO:teuthology.orchestra.run.smithi194.stdout:Access: 2024-03-20 08:38:54.956339801 +0000 2024-03-20T08:38:58.222 INFO:teuthology.orchestra.run.smithi194.stdout:Modify: 2024-03-20 08:38:25.372981812 +0000 2024-03-20T08:38:58.222 INFO:teuthology.orchestra.run.smithi194.stdout:Change: 2024-03-20 08:38:25.372981812 +0000 2024-03-20T08:38:58.222 INFO:teuthology.orchestra.run.smithi194.stdout: Birth: 2024-03-20 08:38:25.372981812 +0000 2024-03-20T08:38:58.222 DEBUG:teuthology.orchestra.run.smithi194:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-03-20T08:38:58.291 INFO:teuthology.orchestra.run.smithi194.stderr:1+0 records in 2024-03-20T08:38:58.291 INFO:teuthology.orchestra.run.smithi194.stderr:1+0 records out 2024-03-20T08:38:58.291 INFO:teuthology.orchestra.run.smithi194.stderr:512 bytes copied, 0.000275282 s, 1.9 MB/s 2024-03-20T08:38:58.292 DEBUG:teuthology.orchestra.run.smithi194:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-03-20T08:38:58.350 DEBUG:teuthology.orchestra.run.smithi194:> stat /dev/vg_nvme/lv_2 2024-03-20T08:38:58.407 INFO:teuthology.orchestra.run.smithi194.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-03-20T08:38:58.407 INFO:teuthology.orchestra.run.smithi194.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-20T08:38:58.407 INFO:teuthology.orchestra.run.smithi194.stdout:Device: 5h/5d Inode: 685 Links: 1 2024-03-20T08:38:58.407 INFO:teuthology.orchestra.run.smithi194.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-20T08:38:58.407 INFO:teuthology.orchestra.run.smithi194.stdout:Context: system_u:object_r:device_t:s0 2024-03-20T08:38:58.408 INFO:teuthology.orchestra.run.smithi194.stdout:Access: 2024-03-20 08:38:54.956339801 +0000 2024-03-20T08:38:58.408 INFO:teuthology.orchestra.run.smithi194.stdout:Modify: 2024-03-20 08:38:25.732973992 +0000 2024-03-20T08:38:58.408 INFO:teuthology.orchestra.run.smithi194.stdout:Change: 2024-03-20 08:38:25.732973992 +0000 2024-03-20T08:38:58.408 INFO:teuthology.orchestra.run.smithi194.stdout: Birth: 2024-03-20 08:38:25.732973992 +0000 2024-03-20T08:38:58.408 DEBUG:teuthology.orchestra.run.smithi194:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-03-20T08:38:58.475 INFO:teuthology.orchestra.run.smithi194.stderr:1+0 records in 2024-03-20T08:38:58.475 INFO:teuthology.orchestra.run.smithi194.stderr:1+0 records out 2024-03-20T08:38:58.475 INFO:teuthology.orchestra.run.smithi194.stderr:512 bytes copied, 0.000200044 s, 2.6 MB/s 2024-03-20T08:38:58.476 DEBUG:teuthology.orchestra.run.smithi194:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-03-20T08:38:58.534 DEBUG:teuthology.orchestra.run.smithi194:> stat /dev/vg_nvme/lv_3 2024-03-20T08:38:58.589 INFO:teuthology.orchestra.run.smithi194.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-03-20T08:38:58.589 INFO:teuthology.orchestra.run.smithi194.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-20T08:38:58.590 INFO:teuthology.orchestra.run.smithi194.stdout:Device: 5h/5d Inode: 705 Links: 1 2024-03-20T08:38:58.590 INFO:teuthology.orchestra.run.smithi194.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-20T08:38:58.590 INFO:teuthology.orchestra.run.smithi194.stdout:Context: system_u:object_r:device_t:s0 2024-03-20T08:38:58.590 INFO:teuthology.orchestra.run.smithi194.stdout:Access: 2024-03-20 08:38:54.956339801 +0000 2024-03-20T08:38:58.590 INFO:teuthology.orchestra.run.smithi194.stdout:Modify: 2024-03-20 08:38:26.086966303 +0000 2024-03-20T08:38:58.590 INFO:teuthology.orchestra.run.smithi194.stdout:Change: 2024-03-20 08:38:26.086966303 +0000 2024-03-20T08:38:58.590 INFO:teuthology.orchestra.run.smithi194.stdout: Birth: 2024-03-20 08:38:26.086966303 +0000 2024-03-20T08:38:58.590 DEBUG:teuthology.orchestra.run.smithi194:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-03-20T08:38:58.656 INFO:teuthology.orchestra.run.smithi194.stderr:1+0 records in 2024-03-20T08:38:58.657 INFO:teuthology.orchestra.run.smithi194.stderr:1+0 records out 2024-03-20T08:38:58.657 INFO:teuthology.orchestra.run.smithi194.stderr:512 bytes copied, 0.000166495 s, 3.1 MB/s 2024-03-20T08:38:58.658 DEBUG:teuthology.orchestra.run.smithi194:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-03-20T08:38:58.714 DEBUG:teuthology.orchestra.run.smithi194:> stat /dev/vg_nvme/lv_4 2024-03-20T08:38:58.770 INFO:teuthology.orchestra.run.smithi194.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-03-20T08:38:58.771 INFO:teuthology.orchestra.run.smithi194.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-20T08:38:58.771 INFO:teuthology.orchestra.run.smithi194.stdout:Device: 5h/5d Inode: 722 Links: 1 2024-03-20T08:38:58.771 INFO:teuthology.orchestra.run.smithi194.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-20T08:38:58.771 INFO:teuthology.orchestra.run.smithi194.stdout:Context: system_u:object_r:device_t:s0 2024-03-20T08:38:58.771 INFO:teuthology.orchestra.run.smithi194.stdout:Access: 2024-03-20 08:38:54.956339801 +0000 2024-03-20T08:38:58.771 INFO:teuthology.orchestra.run.smithi194.stdout:Modify: 2024-03-20 08:38:26.444958529 +0000 2024-03-20T08:38:58.771 INFO:teuthology.orchestra.run.smithi194.stdout:Change: 2024-03-20 08:38:26.444958529 +0000 2024-03-20T08:38:58.771 INFO:teuthology.orchestra.run.smithi194.stdout: Birth: 2024-03-20 08:38:26.444958529 +0000 2024-03-20T08:38:58.771 DEBUG:teuthology.orchestra.run.smithi194:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-03-20T08:38:58.837 INFO:teuthology.orchestra.run.smithi194.stderr:1+0 records in 2024-03-20T08:38:58.837 INFO:teuthology.orchestra.run.smithi194.stderr:1+0 records out 2024-03-20T08:38:58.837 INFO:teuthology.orchestra.run.smithi194.stderr:512 bytes copied, 0.000255821 s, 2.0 MB/s 2024-03-20T08:38:58.838 DEBUG:teuthology.orchestra.run.smithi194:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-03-20T08:38:58.895 DEBUG:teuthology.orchestra.run.smithi194:> grep '^nvme_loop' /proc/modules || sudo modprobe nvme_loop && sudo mkdir -p /sys/kernel/config/nvmet/hosts/hostnqn && sudo mkdir -p /sys/kernel/config/nvmet/ports/1 && echo loop | sudo tee /sys/kernel/config/nvmet/ports/1/addr_trtype 2024-03-20T08:38:59.048 INFO:teuthology.orchestra.run.smithi194.stdout:loop 2024-03-20T08:38:59.049 INFO:tasks.nvme_loop:Connecting nvme_loop smithi194:/dev/vg_nvme/lv_1... 2024-03-20T08:38:59.049 DEBUG:teuthology.orchestra.run.smithi194:> sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_1 && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_1/attr_allow_any_host && sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_1/namespaces/1 && echo -n /dev/vg_nvme/lv_1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_1/namespaces/1/device_path && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_1/namespaces/1/enable && sudo ln -s /sys/kernel/config/nvmet/subsystems/lv_1 /sys/kernel/config/nvmet/ports/1/subsystems/lv_1 && sudo nvme connect -t loop -n lv_1 -q hostnqn 2024-03-20T08:38:59.087 INFO:teuthology.orchestra.run.smithi194.stdout:1 2024-03-20T08:38:59.124 INFO:teuthology.orchestra.run.smithi194.stdout:/dev/vg_nvme/lv_11 2024-03-20T08:38:59.161 INFO:tasks.nvme_loop:Connecting nvme_loop smithi194:/dev/vg_nvme/lv_2... 2024-03-20T08:38:59.161 DEBUG:teuthology.orchestra.run.smithi194:> sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_2 && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_2/attr_allow_any_host && sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_2/namespaces/1 && echo -n /dev/vg_nvme/lv_2 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_2/namespaces/1/device_path && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_2/namespaces/1/enable && sudo ln -s /sys/kernel/config/nvmet/subsystems/lv_2 /sys/kernel/config/nvmet/ports/1/subsystems/lv_2 && sudo nvme connect -t loop -n lv_2 -q hostnqn 2024-03-20T08:38:59.240 INFO:teuthology.orchestra.run.smithi194.stdout:1 2024-03-20T08:38:59.278 INFO:teuthology.orchestra.run.smithi194.stdout:/dev/vg_nvme/lv_21 2024-03-20T08:38:59.305 INFO:tasks.nvme_loop:Connecting nvme_loop smithi194:/dev/vg_nvme/lv_3... 2024-03-20T08:38:59.306 DEBUG:teuthology.orchestra.run.smithi194:> sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_3 && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_3/attr_allow_any_host && sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_3/namespaces/1 && echo -n /dev/vg_nvme/lv_3 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_3/namespaces/1/device_path && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_3/namespaces/1/enable && sudo ln -s /sys/kernel/config/nvmet/subsystems/lv_3 /sys/kernel/config/nvmet/ports/1/subsystems/lv_3 && sudo nvme connect -t loop -n lv_3 -q hostnqn 2024-03-20T08:38:59.384 INFO:teuthology.orchestra.run.smithi194.stdout:1 2024-03-20T08:38:59.422 INFO:teuthology.orchestra.run.smithi194.stdout:/dev/vg_nvme/lv_31 2024-03-20T08:38:59.450 INFO:tasks.nvme_loop:Connecting nvme_loop smithi194:/dev/vg_nvme/lv_4... 2024-03-20T08:38:59.450 DEBUG:teuthology.orchestra.run.smithi194:> sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_4 && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_4/attr_allow_any_host && sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_4/namespaces/1 && echo -n /dev/vg_nvme/lv_4 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_4/namespaces/1/device_path && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_4/namespaces/1/enable && sudo ln -s /sys/kernel/config/nvmet/subsystems/lv_4 /sys/kernel/config/nvmet/ports/1/subsystems/lv_4 && sudo nvme connect -t loop -n lv_4 -q hostnqn 2024-03-20T08:38:59.528 INFO:teuthology.orchestra.run.smithi194.stdout:1 2024-03-20T08:38:59.565 INFO:teuthology.orchestra.run.smithi194.stdout:/dev/vg_nvme/lv_41 2024-03-20T08:38:59.593 DEBUG:teuthology.orchestra.run.smithi194:> set -ex 2024-03-20T08:38:59.593 DEBUG:teuthology.orchestra.run.smithi194:> dd if=/scratch_devs of=/dev/stdout 2024-03-20T08:38:59.649 DEBUG:teuthology.orchestra.run.smithi194:> sudo nvme list 2024-03-20T08:38:59.714 INFO:teuthology.orchestra.run.smithi194.stdout:Node SN Model Namespace Usage Format FW Rev 2024-03-20T08:38:59.715 INFO:teuthology.orchestra.run.smithi194.stdout:---------------- -------------------- ---------------------------------------- --------- -------------------------- ---------------- -------- 2024-03-20T08:38:59.715 INFO:teuthology.orchestra.run.smithi194.stdout:/dev/nvme0n1 CVFT6240002A400BGN INTEL SSDPEDMD400G4 1 400.09 GB / 400.09 GB 512 B + 0 B 8DV101H0 2024-03-20T08:38:59.715 INFO:teuthology.orchestra.run.smithi194.stdout:/dev/nvme1n1 9235e01bc4fe137c7840 Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 5.14.0-4 2024-03-20T08:38:59.715 INFO:teuthology.orchestra.run.smithi194.stdout:/dev/nvme2n1 71a165ac3a14b88708e4 Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 5.14.0-4 2024-03-20T08:38:59.715 INFO:teuthology.orchestra.run.smithi194.stdout:/dev/nvme3n1 009bba8792f40b5d0ea3 Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 5.14.0-4 2024-03-20T08:38:59.715 INFO:teuthology.orchestra.run.smithi194.stdout:/dev/nvme4n1 d8dd6ecf1af39479d860 Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 5.14.0-4 2024-03-20T08:38:59.716 INFO:tasks.nvme_loop:new_devs ['/dev/nvme1n1', '/dev/nvme2n1', '/dev/nvme3n1', '/dev/nvme4n1'] 2024-03-20T08:38:59.716 DEBUG:teuthology.orchestra.run.smithi194:> set -ex 2024-03-20T08:38:59.716 DEBUG:teuthology.orchestra.run.smithi194:> sudo dd of=/scratch_devs 2024-03-20T08:38:59.783 INFO:teuthology.run_tasks:Running task cephadm... 2024-03-20T08:38:59.878 INFO:tasks.cephadm:Config: {'conf': {'mgr': {'debug mgr': 20, 'debug ms': 1, 'mgr/cephadm/use_agent': True}, 'global': {'mon election default strategy': 3}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'debug ms': 1, 'debug osd': 20, 'osd shutdown pgref assert': True}}, 'flavor': 'default', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'MON_DOWN'], 'sha1': 'e7e80a6ac9849f0f23432d0d9832f096730308ae'} 2024-03-20T08:38:59.878 INFO:tasks.cephadm:Cluster image is quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae 2024-03-20T08:38:59.879 INFO:tasks.cephadm:Cluster fsid is 4c3f3eba-e695-11ee-95c9-87774f69a715 2024-03-20T08:38:59.879 INFO:tasks.cephadm:Choosing monitor IPs and ports... 2024-03-20T08:38:59.879 INFO:tasks.cephadm:Monitor IPs: {'mon.a': '172.21.15.183', 'mon.c': '[v2:172.21.15.183:3301,v1:172.21.15.183:6790]', 'mon.b': '172.21.15.194'} 2024-03-20T08:38:59.879 INFO:tasks.cephadm:First mon is mon.a on smithi183 2024-03-20T08:38:59.879 INFO:tasks.cephadm:First mgr is y 2024-03-20T08:38:59.879 INFO:tasks.cephadm:Normalizing hostnames... 2024-03-20T08:38:59.880 DEBUG:teuthology.orchestra.run.smithi183:> sudo hostname $(hostname -s) 2024-03-20T08:38:59.912 DEBUG:teuthology.orchestra.run.smithi194:> sudo hostname $(hostname -s) 2024-03-20T08:38:59.939 INFO:tasks.cephadm:Downloading "compiled" cephadm from cachra 2024-03-20T08:38:59.940 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=e7e80a6ac9849f0f23432d0d9832f096730308ae 2024-03-20T08:39:00.379 INFO:tasks.cephadm:builder_project result: [{'status': 'ready', 'sha1': 'e7e80a6ac9849f0f23432d0d9832f096730308ae', 'extra': {'build_url': 'https://jenkins.ceph.com/job/ceph-dev-new-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=centos9,DIST=centos9,MACHINE_SIZE=gigantic/77739/', 'root_build_cause': 'SCMTRIGGER', 'version': '19.0.0-2153-ge7e80a6a', 'node_name': '172.21.5.36+adami06', 'job_name': 'ceph-dev-new-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=centos9,DIST=centos9,MACHINE_SIZE=gigantic', 'package_manager_version': '19.0.0-2153.ge7e80a6a'}, 'url': 'https://3.chacra.ceph.com/r/ceph/wip-yuri5-testing-2024-03-18-1144/e7e80a6ac9849f0f23432d0d9832f096730308ae/centos/9/flavors/default/', 'distro_codename': None, 'modified': '2024-03-18 20:22:32.637670', 'distro_version': '9', 'project': 'ceph', 'flavor': 'default', 'ref': 'wip-yuri5-testing-2024-03-18-1144', 'chacra_url': 'https://3.chacra.ceph.com/repos/ceph/wip-yuri5-testing-2024-03-18-1144/e7e80a6ac9849f0f23432d0d9832f096730308ae/centos/9/flavors/default/', 'archs': ['x86_64', 'source'], 'distro': 'centos'}] 2024-03-20T08:39:00.519 INFO:tasks.util.chacra:got chacra host 3.chacra.ceph.com, ref wip-yuri5-testing-2024-03-18-1144, sha1 e7e80a6ac9849f0f23432d0d9832f096730308ae from https://shaman.ceph.com/api/search/?project=ceph&distros=centos%2F9%2Fx86_64&flavor=default&sha1=e7e80a6ac9849f0f23432d0d9832f096730308ae 2024-03-20T08:39:00.521 INFO:tasks.cephadm:Discovered cachra url: https://3.chacra.ceph.com/binaries/ceph/wip-yuri5-testing-2024-03-18-1144/e7e80a6ac9849f0f23432d0d9832f096730308ae/centos/9/x86_64/flavors/default/cephadm 2024-03-20T08:39:00.521 DEBUG:teuthology.orchestra.run.smithi183:> curl --silent -L https://3.chacra.ceph.com/binaries/ceph/wip-yuri5-testing-2024-03-18-1144/e7e80a6ac9849f0f23432d0d9832f096730308ae/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-03-20T08:39:00.971 INFO:teuthology.orchestra.run.smithi183.stdout:-rw-r--r--. 1 ubuntu ubuntu 767380 Mar 20 08:39 /home/ubuntu/cephtest/cephadm 2024-03-20T08:39:00.971 DEBUG:teuthology.orchestra.run.smithi194:> curl --silent -L https://3.chacra.ceph.com/binaries/ceph/wip-yuri5-testing-2024-03-18-1144/e7e80a6ac9849f0f23432d0d9832f096730308ae/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-03-20T08:39:01.416 INFO:teuthology.orchestra.run.smithi194.stdout:-rw-r--r--. 1 ubuntu ubuntu 767380 Mar 20 08:39 /home/ubuntu/cephtest/cephadm 2024-03-20T08:39:01.417 DEBUG:teuthology.orchestra.run.smithi183:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-03-20T08:39:01.435 DEBUG:teuthology.orchestra.run.smithi194:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-03-20T08:39:01.465 INFO:tasks.cephadm:Pulling image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae on all hosts... 2024-03-20T08:39:01.465 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae pull 2024-03-20T08:39:01.478 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae pull 2024-03-20T08:39:02.240 INFO:teuthology.orchestra.run.smithi194.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae... 2024-03-20T08:39:02.308 INFO:teuthology.orchestra.run.smithi183.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae... 2024-03-20T08:39:36.601 INFO:teuthology.orchestra.run.smithi183.stdout:{ 2024-03-20T08:39:36.602 INFO:teuthology.orchestra.run.smithi183.stdout: "ceph_version": "ceph version 19.0.0-2153-ge7e80a6a (e7e80a6ac9849f0f23432d0d9832f096730308ae) squid (dev)", 2024-03-20T08:39:36.602 INFO:teuthology.orchestra.run.smithi183.stdout: "image_id": "9ceba81fb2c8d8682cb159527017d591637b153b3aa4a390ffa8586db6dcda00", 2024-03-20T08:39:36.602 INFO:teuthology.orchestra.run.smithi183.stdout: "repo_digests": [ 2024-03-20T08:39:36.602 INFO:teuthology.orchestra.run.smithi183.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:976a8b2d06e3c6bfd12488297e8584dd704cc06c3e976c7e0f718c13e238e912" 2024-03-20T08:39:36.602 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-03-20T08:39:36.602 INFO:teuthology.orchestra.run.smithi183.stdout:} 2024-03-20T08:39:50.247 INFO:teuthology.orchestra.run.smithi194.stdout:{ 2024-03-20T08:39:50.247 INFO:teuthology.orchestra.run.smithi194.stdout: "ceph_version": "ceph version 19.0.0-2153-ge7e80a6a (e7e80a6ac9849f0f23432d0d9832f096730308ae) squid (dev)", 2024-03-20T08:39:50.247 INFO:teuthology.orchestra.run.smithi194.stdout: "image_id": "9ceba81fb2c8d8682cb159527017d591637b153b3aa4a390ffa8586db6dcda00", 2024-03-20T08:39:50.247 INFO:teuthology.orchestra.run.smithi194.stdout: "repo_digests": [ 2024-03-20T08:39:50.248 INFO:teuthology.orchestra.run.smithi194.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:976a8b2d06e3c6bfd12488297e8584dd704cc06c3e976c7e0f718c13e238e912" 2024-03-20T08:39:50.248 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-03-20T08:39:50.248 INFO:teuthology.orchestra.run.smithi194.stdout:} 2024-03-20T08:39:50.276 DEBUG:teuthology.orchestra.run.smithi183:> sudo mkdir -p /etc/ceph 2024-03-20T08:39:50.311 DEBUG:teuthology.orchestra.run.smithi194:> sudo mkdir -p /etc/ceph 2024-03-20T08:39:50.342 DEBUG:teuthology.orchestra.run.smithi183:> sudo chmod 777 /etc/ceph 2024-03-20T08:39:50.380 DEBUG:teuthology.orchestra.run.smithi194:> sudo chmod 777 /etc/ceph 2024-03-20T08:39:50.412 INFO:tasks.cephadm:Writing seed config... 2024-03-20T08:39:50.413 INFO:tasks.cephadm: override: [mgr] debug mgr = 20 2024-03-20T08:39:50.413 INFO:tasks.cephadm: override: [mgr] debug ms = 1 2024-03-20T08:39:50.413 INFO:tasks.cephadm: override: [mgr] mgr/cephadm/use_agent = True 2024-03-20T08:39:50.414 INFO:tasks.cephadm: override: [global] mon election default strategy = 3 2024-03-20T08:39:50.414 INFO:tasks.cephadm: override: [mon] debug mon = 20 2024-03-20T08:39:50.414 INFO:tasks.cephadm: override: [mon] debug ms = 1 2024-03-20T08:39:50.414 INFO:tasks.cephadm: override: [mon] debug paxos = 20 2024-03-20T08:39:50.414 INFO:tasks.cephadm: override: [osd] debug ms = 1 2024-03-20T08:39:50.414 INFO:tasks.cephadm: override: [osd] debug osd = 20 2024-03-20T08:39:50.414 INFO:tasks.cephadm: override: [osd] osd shutdown pgref assert = True 2024-03-20T08:39:50.415 DEBUG:teuthology.orchestra.run.smithi183:> set -ex 2024-03-20T08:39:50.415 DEBUG:teuthology.orchestra.run.smithi183:> dd of=/home/ubuntu/cephtest/seed.ceph.conf 2024-03-20T08:39:50.436 DEBUG:tasks.cephadm:Final config: [global] # make logging friendly to teuthology log_to_file = true log_to_stderr = false log to journald = false mon cluster log to file = true mon cluster log file level = debug mon clock drift allowed = 1.000 # replicate across OSDs, not hosts osd crush chooseleaf type = 0 #osd pool default size = 2 osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd # enable some debugging auth debug = true ms die on old message = true ms die on bug = true debug asserts on shutdown = true # adjust warnings mon max pg per osd = 10000# >= luminous mon pg warn max object skew = 0 mon osd allow primary affinity = true mon osd allow pg remap = true mon warn on legacy crush tunables = false mon warn on crush straw calc version zero = false mon warn on no sortbitwise = false mon warn on osd down out interval zero = false mon warn on too few osds = false mon_warn_on_pool_pg_num_not_power_of_two = false # disable pg_autoscaler by default for new pools osd_pool_default_pg_autoscale_mode = off # tests delete pools mon allow pool delete = true fsid = 4c3f3eba-e695-11ee-95c9-87774f69a715 mon election default strategy = 3 [osd] osd scrub load threshold = 5.0 osd scrub max interval = 600 osd mclock profile = high_recovery_ops osd recover clone overlap = true osd recovery max chunk = 1048576 osd deep scrub update digest min age = 30 osd map max advance = 10 osd memory target autotune = true # debugging osd debug shutdown = true osd debug op order = true osd debug verify stray on activate = true osd debug pg log writeout = true osd debug verify cached snaps = true osd debug verify missing on start = true osd debug misdirected ops = true osd op queue = debug_random osd op queue cut off = debug_random osd shutdown pgref assert = True bdev debug aio = true osd sloppy crc = true debug ms = 1 debug osd = 20 [mgr] mon reweight min pgs per osd = 4 mon reweight min bytes per osd = 10 mgr/telemetry/nag = false debug mgr = 20 debug ms = 1 mgr/cephadm/use_agent = True [mon] mon data avail warn = 5 mon mgr mkfs grace = 240 mon reweight min pgs per osd = 4 mon osd reporter subtree level = osd mon osd prime pg temp = true mon reweight min bytes per osd = 10 # rotate auth tickets quickly to exercise renewal paths auth mon ticket ttl = 660# 11m auth service ticket ttl = 240# 4m # don't complain about global id reclaim mon_warn_on_insecure_global_id_reclaim = false mon_warn_on_insecure_global_id_reclaim_allowed = false debug mon = 20 debug ms = 1 debug paxos = 20 [client.rgw] rgw cache enabled = true rgw enable ops log = true rgw enable usage log = true 2024-03-20T08:39:50.437 DEBUG:teuthology.orchestra.run.smithi183:mon.a> sudo journalctl -f -n 0 -u ceph-4c3f3eba-e695-11ee-95c9-87774f69a715@mon.a.service 2024-03-20T08:39:50.479 DEBUG:teuthology.orchestra.run.smithi183:mgr.y> sudo journalctl -f -n 0 -u ceph-4c3f3eba-e695-11ee-95c9-87774f69a715@mgr.y.service 2024-03-20T08:39:50.521 INFO:tasks.cephadm:Bootstrapping... 2024-03-20T08:39:50.522 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae -v bootstrap --fsid 4c3f3eba-e695-11ee-95c9-87774f69a715 --config /home/ubuntu/cephtest/seed.ceph.conf --output-config /etc/ceph/ceph.conf --output-keyring /etc/ceph/ceph.client.admin.keyring --output-pub-ssh-key /home/ubuntu/cephtest/ceph.pub --mon-id a --mgr-id y --orphan-initial-daemons --skip-monitoring-stack --mon-ip 172.21.15.183 --skip-admin-label && sudo chmod +r /etc/ceph/ceph.client.admin.keyring 2024-03-20T08:39:50.715 INFO:teuthology.orchestra.run.smithi183.stdout:-------------------------------------------------------------------------------- 2024-03-20T08:39:50.715 INFO:teuthology.orchestra.run.smithi183.stdout:cephadm ['--image', 'quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae', '-v', 'bootstrap', '--fsid', '4c3f3eba-e695-11ee-95c9-87774f69a715', '--config', '/home/ubuntu/cephtest/seed.ceph.conf', '--output-config', '/etc/ceph/ceph.conf', '--output-keyring', '/etc/ceph/ceph.client.admin.keyring', '--output-pub-ssh-key', '/home/ubuntu/cephtest/ceph.pub', '--mon-id', 'a', '--mgr-id', 'y', '--orphan-initial-daemons', '--skip-monitoring-stack', '--mon-ip', '172.21.15.183', '--skip-admin-label'] 2024-03-20T08:39:50.739 INFO:teuthology.orchestra.run.smithi183.stdout:/bin/podman: stdout 4.9.3-dev 2024-03-20T08:39:50.740 INFO:teuthology.orchestra.run.smithi183.stderr:Specifying an fsid for your cluster offers no advantages and may increase the likelihood of fsid conflicts. 2024-03-20T08:39:50.740 INFO:teuthology.orchestra.run.smithi183.stdout:Verifying podman|docker is present... 2024-03-20T08:39:50.763 INFO:teuthology.orchestra.run.smithi183.stdout:/bin/podman: stdout 4.9.3-dev 2024-03-20T08:39:50.764 INFO:teuthology.orchestra.run.smithi183.stdout:Verifying lvm2 is present... 2024-03-20T08:39:50.764 INFO:teuthology.orchestra.run.smithi183.stdout:Verifying time synchronization is in place... 2024-03-20T08:39:50.770 INFO:teuthology.orchestra.run.smithi183.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-03-20T08:39:50.770 INFO:teuthology.orchestra.run.smithi183.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-03-20T08:39:50.776 INFO:teuthology.orchestra.run.smithi183.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-03-20T08:39:50.776 INFO:teuthology.orchestra.run.smithi183.stdout:systemctl: stdout inactive 2024-03-20T08:39:50.782 INFO:teuthology.orchestra.run.smithi183.stdout:systemctl: stdout enabled 2024-03-20T08:39:50.788 INFO:teuthology.orchestra.run.smithi183.stdout:systemctl: stdout active 2024-03-20T08:39:50.788 INFO:teuthology.orchestra.run.smithi183.stdout:Unit chronyd.service is enabled and running 2024-03-20T08:39:50.788 INFO:teuthology.orchestra.run.smithi183.stdout:Repeating the final host check... 2024-03-20T08:39:50.811 INFO:teuthology.orchestra.run.smithi183.stdout:/bin/podman: stdout 4.9.3-dev 2024-03-20T08:39:50.811 INFO:teuthology.orchestra.run.smithi183.stdout:podman (/bin/podman) version 4.9.3 is present 2024-03-20T08:39:50.812 INFO:teuthology.orchestra.run.smithi183.stdout:systemctl is present 2024-03-20T08:39:50.812 INFO:teuthology.orchestra.run.smithi183.stdout:lvcreate is present 2024-03-20T08:39:50.818 INFO:teuthology.orchestra.run.smithi183.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-03-20T08:39:50.818 INFO:teuthology.orchestra.run.smithi183.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-03-20T08:39:50.824 INFO:teuthology.orchestra.run.smithi183.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-03-20T08:39:50.824 INFO:teuthology.orchestra.run.smithi183.stdout:systemctl: stdout inactive 2024-03-20T08:39:50.830 INFO:teuthology.orchestra.run.smithi183.stdout:systemctl: stdout enabled 2024-03-20T08:39:50.836 INFO:teuthology.orchestra.run.smithi183.stdout:systemctl: stdout active 2024-03-20T08:39:50.836 INFO:teuthology.orchestra.run.smithi183.stdout:Unit chronyd.service is enabled and running 2024-03-20T08:39:50.836 INFO:teuthology.orchestra.run.smithi183.stdout:Host looks OK 2024-03-20T08:39:50.836 INFO:teuthology.orchestra.run.smithi183.stdout:Cluster fsid: 4c3f3eba-e695-11ee-95c9-87774f69a715 2024-03-20T08:39:50.836 INFO:teuthology.orchestra.run.smithi183.stdout:Acquiring lock 140502963680640 on /run/cephadm/4c3f3eba-e695-11ee-95c9-87774f69a715.lock 2024-03-20T08:39:50.837 INFO:teuthology.orchestra.run.smithi183.stdout:Lock 140502963680640 acquired on /run/cephadm/4c3f3eba-e695-11ee-95c9-87774f69a715.lock 2024-03-20T08:39:50.837 INFO:teuthology.orchestra.run.smithi183.stdout:Verifying IP 172.21.15.183 port 3300 ... 2024-03-20T08:39:50.837 INFO:teuthology.orchestra.run.smithi183.stdout:Verifying IP 172.21.15.183 port 6789 ... 2024-03-20T08:39:50.837 INFO:teuthology.orchestra.run.smithi183.stdout:Base mon IP(s) is [172.21.15.183:3300, 172.21.15.183:6789], mon addrv is [v2:172.21.15.183:3300,v1:172.21.15.183:6789] 2024-03-20T08:39:50.840 INFO:teuthology.orchestra.run.smithi183.stdout:/sbin/ip: stdout default via 172.21.15.254 dev enp3s0f1 proto dhcp src 172.21.15.183 metric 100 2024-03-20T08:39:50.840 INFO:teuthology.orchestra.run.smithi183.stdout:/sbin/ip: stdout 172.21.0.0/20 dev enp3s0f1 proto kernel scope link src 172.21.15.183 metric 100 2024-03-20T08:39:50.843 INFO:teuthology.orchestra.run.smithi183.stdout:/sbin/ip: stdout ::1 dev lo proto kernel metric 256 pref medium 2024-03-20T08:39:50.843 INFO:teuthology.orchestra.run.smithi183.stdout:/sbin/ip: stdout fe80::/64 dev enp3s0f1 proto kernel metric 256 pref medium 2024-03-20T08:39:50.843 INFO:teuthology.orchestra.run.smithi183.stdout:/sbin/ip: stdout default via fe80::327c:5e00:6487:71e0 dev enp3s0f1 proto ra metric 1024 expires 1792sec hoplimit 64 pref medium 2024-03-20T08:39:50.845 INFO:teuthology.orchestra.run.smithi183.stdout:/sbin/ip: stdout 1: lo: mtu 65536 state UNKNOWN qlen 1000 2024-03-20T08:39:50.845 INFO:teuthology.orchestra.run.smithi183.stdout:/sbin/ip: stdout inet6 ::1/128 scope host 2024-03-20T08:39:50.846 INFO:teuthology.orchestra.run.smithi183.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-03-20T08:39:50.846 INFO:teuthology.orchestra.run.smithi183.stdout:/sbin/ip: stdout 5: enp3s0f1: mtu 1500 state UP qlen 1000 2024-03-20T08:39:50.846 INFO:teuthology.orchestra.run.smithi183.stdout:/sbin/ip: stdout inet6 fe80::ec4:7aff:fe88:8085/64 scope link 2024-03-20T08:39:50.846 INFO:teuthology.orchestra.run.smithi183.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-03-20T08:39:50.846 INFO:teuthology.orchestra.run.smithi183.stdout:Mon IP `172.21.15.183` is in CIDR network `172.21.0.0/20` 2024-03-20T08:39:50.846 INFO:teuthology.orchestra.run.smithi183.stdout:Mon IP `172.21.15.183` is in CIDR network `172.21.0.0/20` 2024-03-20T08:39:50.847 INFO:teuthology.orchestra.run.smithi183.stdout:Inferred mon public CIDR from local network configuration ['172.21.0.0/20', '172.21.0.0/20'] 2024-03-20T08:39:50.847 INFO:teuthology.orchestra.run.smithi183.stdout:Internal network (--cluster-network) has not been provided, OSD replication will default to the public_network 2024-03-20T08:39:50.847 INFO:teuthology.orchestra.run.smithi183.stdout:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae... 2024-03-20T08:39:51.344 INFO:teuthology.orchestra.run.smithi183.stdout:/bin/podman: stdout 9ceba81fb2c8d8682cb159527017d591637b153b3aa4a390ffa8586db6dcda00 2024-03-20T08:39:51.344 INFO:teuthology.orchestra.run.smithi183.stdout:/bin/podman: stderr Trying to pull quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae... 2024-03-20T08:39:51.344 INFO:teuthology.orchestra.run.smithi183.stdout:/bin/podman: stderr Getting image source signatures 2024-03-20T08:39:51.344 INFO:teuthology.orchestra.run.smithi183.stdout:/bin/podman: stderr Copying blob sha256:29f2579974f82cd19c8183e36ca34a7e5a6e57e8b4789a8e115604a69a0d38b4 2024-03-20T08:39:51.344 INFO:teuthology.orchestra.run.smithi183.stdout:/bin/podman: stderr Copying blob sha256:f58b4670ef4403d11d1c912aa26a4dbcf73447df5302e9dc66f6a7fc0d9d1e47 2024-03-20T08:39:51.344 INFO:teuthology.orchestra.run.smithi183.stdout:/bin/podman: stderr Copying config sha256:9ceba81fb2c8d8682cb159527017d591637b153b3aa4a390ffa8586db6dcda00 2024-03-20T08:39:51.344 INFO:teuthology.orchestra.run.smithi183.stdout:/bin/podman: stderr Writing manifest to image destination 2024-03-20T08:39:52.222 INFO:teuthology.orchestra.run.smithi183.stdout:ceph: stdout ceph version 19.0.0-2153-ge7e80a6a (e7e80a6ac9849f0f23432d0d9832f096730308ae) squid (dev) 2024-03-20T08:39:52.222 INFO:teuthology.orchestra.run.smithi183.stdout:Ceph version: ceph version 19.0.0-2153-ge7e80a6a (e7e80a6ac9849f0f23432d0d9832f096730308ae) squid (dev) 2024-03-20T08:39:52.222 INFO:teuthology.orchestra.run.smithi183.stdout:Extracting ceph user uid/gid from container image... 2024-03-20T08:39:53.228 INFO:teuthology.orchestra.run.smithi183.stdout:stat: stdout 167 167 2024-03-20T08:39:53.228 INFO:teuthology.orchestra.run.smithi183.stdout:Creating initial keys... 2024-03-20T08:39:54.050 INFO:teuthology.orchestra.run.smithi183.stdout:/usr/bin/ceph-authtool: stdout AQDZoPpla6+PIBAAjnwWr5Hxq18gnMSxKpWS1A== 2024-03-20T08:39:55.064 INFO:teuthology.orchestra.run.smithi183.stdout:/usr/bin/ceph-authtool: stdout AQDaoPplos8FGRAAjDkoiVJ+BG6fPVVqFt8bEQ== 2024-03-20T08:39:56.097 INFO:teuthology.orchestra.run.smithi183.stdout:/usr/bin/ceph-authtool: stdout AQDboPplhwdLGBAAYcph7424Cgz1tlFtvqfscQ== 2024-03-20T08:39:56.098 INFO:teuthology.orchestra.run.smithi183.stdout:Creating initial monmap... 2024-03-20T08:39:56.976 INFO:teuthology.orchestra.run.smithi183.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: monmap file /tmp/monmap 2024-03-20T08:39:56.976 INFO:teuthology.orchestra.run.smithi183.stdout:/usr/bin/monmaptool: stdout setting min_mon_release = quincy 2024-03-20T08:39:56.976 INFO:teuthology.orchestra.run.smithi183.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: set fsid to 4c3f3eba-e695-11ee-95c9-87774f69a715 2024-03-20T08:39:56.976 INFO:teuthology.orchestra.run.smithi183.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-03-20T08:39:56.977 INFO:teuthology.orchestra.run.smithi183.stdout:monmaptool for a [v2:172.21.15.183:3300,v1:172.21.15.183:6789] on /usr/bin/monmaptool: monmap file /tmp/monmap 2024-03-20T08:39:56.977 INFO:teuthology.orchestra.run.smithi183.stdout:setting min_mon_release = quincy 2024-03-20T08:39:56.977 INFO:teuthology.orchestra.run.smithi183.stdout:/usr/bin/monmaptool: set fsid to 4c3f3eba-e695-11ee-95c9-87774f69a715 2024-03-20T08:39:56.977 INFO:teuthology.orchestra.run.smithi183.stdout:/usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-03-20T08:39:56.977 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-20T08:39:56.977 INFO:teuthology.orchestra.run.smithi183.stdout:Creating mon... 2024-03-20T08:39:59.056 INFO:teuthology.orchestra.run.smithi183.stdout:create mon.a on 2024-03-20T08:39:59.751 INFO:teuthology.orchestra.run.smithi183.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /etc/systemd/system/ceph.target. 2024-03-20T08:40:00.121 INFO:teuthology.orchestra.run.smithi183.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph-4c3f3eba-e695-11ee-95c9-87774f69a715.target → /etc/systemd/system/ceph-4c3f3eba-e695-11ee-95c9-87774f69a715.target. 2024-03-20T08:40:00.122 INFO:teuthology.orchestra.run.smithi183.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph.target.wants/ceph-4c3f3eba-e695-11ee-95c9-87774f69a715.target → /etc/systemd/system/ceph-4c3f3eba-e695-11ee-95c9-87774f69a715.target. 2024-03-20T08:40:00.784 INFO:teuthology.orchestra.run.smithi183.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-4c3f3eba-e695-11ee-95c9-87774f69a715@mon.a 2024-03-20T08:40:00.784 INFO:teuthology.orchestra.run.smithi183.stdout:systemctl: stderr Failed to reset failed state of unit ceph-4c3f3eba-e695-11ee-95c9-87774f69a715@mon.a.service: Unit ceph-4c3f3eba-e695-11ee-95c9-87774f69a715@mon.a.service not loaded. 2024-03-20T08:40:01.022 INFO:teuthology.orchestra.run.smithi183.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-4c3f3eba-e695-11ee-95c9-87774f69a715.target.wants/ceph-4c3f3eba-e695-11ee-95c9-87774f69a715@mon.a.service → /etc/systemd/system/ceph-4c3f3eba-e695-11ee-95c9-87774f69a715@.service. 2024-03-20T08:40:01.454 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:40:01 smithi183 podman[24882]: 2024-03-20 08:40:01.159054299 +0000 UTC m=+0.017622320 image pull 9ceba81fb2c8d8682cb159527017d591637b153b3aa4a390ffa8586db6dcda00 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae 2024-03-20T08:40:02.704 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:40:02 smithi183 podman[24882]: 2024-03-20 08:40:02.445851482 +0000 UTC m=+1.304419541 container create 1e0acaacccdaf0316251161831509e42513469c5ae9fb6f526ebb7dc684b7d20 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae, name=ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-mon-a, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, RELEASE=wip-yuri5-testing-2024-03-18-1144-e7e80a6, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, org.label-schema.build-date=20240307, GIT_CLEAN=True, maintainer=Guillaume Abrioux ) 2024-03-20T08:40:04.704 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:40:04 smithi183 podman[24882]: 2024-03-20 08:40:04.253667718 +0000 UTC m=+3.112235744 container init 1e0acaacccdaf0316251161831509e42513469c5ae9fb6f526ebb7dc684b7d20 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae, name=ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-mon-a, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-yuri5-testing-2024-03-18-1144-e7e80a6, GIT_CLEAN=True, org.label-schema.build-date=20240307, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=) 2024-03-20T08:40:04.704 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:40:04 smithi183 podman[24882]: 2024-03-20 08:40:04.257622517 +0000 UTC m=+3.116190540 container start 1e0acaacccdaf0316251161831509e42513469c5ae9fb6f526ebb7dc684b7d20 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae, name=ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-mon-a, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, RELEASE=wip-yuri5-testing-2024-03-18-1144-e7e80a6, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240307, CEPH_POINT_RELEASE=, GIT_CLEAN=True) 2024-03-20T08:40:04.705 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:40:04 smithi183 ceph-mon[24901]: mkfs 4c3f3eba-e695-11ee-95c9-87774f69a715 2024-03-20T08:40:04.705 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:40:04 smithi183 ceph-mon[24901]: mon.a is new leader, mons a in quorum (ranks 0) 2024-03-20T08:40:05.172 INFO:teuthology.orchestra.run.smithi183.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-03-20T08:40:05.172 INFO:teuthology.orchestra.run.smithi183.stdout:systemctl: stdout disabled 2024-03-20T08:40:05.180 INFO:teuthology.orchestra.run.smithi183.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-03-20T08:40:05.180 INFO:teuthology.orchestra.run.smithi183.stdout:systemctl: stdout inactive 2024-03-20T08:40:05.180 INFO:teuthology.orchestra.run.smithi183.stdout:firewalld.service is not enabled 2024-03-20T08:40:05.180 INFO:teuthology.orchestra.run.smithi183.stdout:Not possible to enable service . firewalld.service is not available 2024-03-20T08:40:05.181 INFO:teuthology.orchestra.run.smithi183.stdout:Waiting for mon to start... 2024-03-20T08:40:05.181 INFO:teuthology.orchestra.run.smithi183.stdout:Waiting for mon... 2024-03-20T08:40:05.457 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:40:05 smithi183 bash[24882]: 1e0acaacccdaf0316251161831509e42513469c5ae9fb6f526ebb7dc684b7d20 2024-03-20T08:40:05.457 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:40:05 smithi183 systemd[1]: Started Ceph mon.a for 4c3f3eba-e695-11ee-95c9-87774f69a715. 2024-03-20T08:40:05.457 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:40:05 smithi183 ceph-mon[24901]: mon.a is new leader, mons a in quorum (ranks 0) 2024-03-20T08:40:05.458 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:40:05 smithi183 ceph-mon[24901]: monmap e1: 1 mons at {a=[v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-20T08:40:05.458 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:40:05 smithi183 ceph-mon[24901]: fsmap 2024-03-20T08:40:05.458 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:40:05 smithi183 ceph-mon[24901]: osdmap e1: 0 total, 0 up, 0 in 2024-03-20T08:40:05.458 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:40:05 smithi183 ceph-mon[24901]: mgrmap e1: no daemons active 2024-03-20T08:40:06.488 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:40:06 smithi183 ceph-mon[24901]: from='client.? 172.21.15.183:0/1424182065' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch 2024-03-20T08:40:06.492 INFO:teuthology.orchestra.run.smithi183.stdout:/usr/bin/ceph: stdout cluster: 2024-03-20T08:40:06.492 INFO:teuthology.orchestra.run.smithi183.stdout:/usr/bin/ceph: stdout id: 4c3f3eba-e695-11ee-95c9-87774f69a715 2024-03-20T08:40:06.492 INFO:teuthology.orchestra.run.smithi183.stdout:/usr/bin/ceph: stdout health: HEALTH_OK 2024-03-20T08:40:06.492 INFO:teuthology.orchestra.run.smithi183.stdout:/usr/bin/ceph: stdout 2024-03-20T08:40:06.492 INFO:teuthology.orchestra.run.smithi183.stdout:/usr/bin/ceph: stdout services: 2024-03-20T08:40:06.493 INFO:teuthology.orchestra.run.smithi183.stdout:/usr/bin/ceph: stdout mon: 1 daemons, quorum a (age 1.63677s) 2024-03-20T08:40:06.493 INFO:teuthology.orchestra.run.smithi183.stdout:/usr/bin/ceph: stdout mgr: no daemons active 2024-03-20T08:40:06.493 INFO:teuthology.orchestra.run.smithi183.stdout:/usr/bin/ceph: stdout osd: 0 osds: 0 up, 0 in 2024-03-20T08:40:06.493 INFO:teuthology.orchestra.run.smithi183.stdout:/usr/bin/ceph: stdout 2024-03-20T08:40:06.493 INFO:teuthology.orchestra.run.smithi183.stdout:/usr/bin/ceph: stdout data: 2024-03-20T08:40:06.493 INFO:teuthology.orchestra.run.smithi183.stdout:/usr/bin/ceph: stdout pools: 0 pools, 0 pgs 2024-03-20T08:40:06.493 INFO:teuthology.orchestra.run.smithi183.stdout:/usr/bin/ceph: stdout objects: 0 objects, 0 B 2024-03-20T08:40:06.493 INFO:teuthology.orchestra.run.smithi183.stdout:/usr/bin/ceph: stdout usage: 0 B used, 0 B / 0 B avail 2024-03-20T08:40:06.493 INFO:teuthology.orchestra.run.smithi183.stdout:/usr/bin/ceph: stdout pgs: 2024-03-20T08:40:06.493 INFO:teuthology.orchestra.run.smithi183.stdout:/usr/bin/ceph: stdout 2024-03-20T08:40:06.493 INFO:teuthology.orchestra.run.smithi183.stdout:mon is available 2024-03-20T08:40:06.493 INFO:teuthology.orchestra.run.smithi183.stdout:Assimilating anything we can from ceph.conf... 2024-03-20T08:40:07.457 INFO:teuthology.orchestra.run.smithi183.stdout:/usr/bin/ceph: stdout 2024-03-20T08:40:07.457 INFO:teuthology.orchestra.run.smithi183.stdout:/usr/bin/ceph: stdout [global] 2024-03-20T08:40:07.457 INFO:teuthology.orchestra.run.smithi183.stdout:/usr/bin/ceph: stdout fsid = 4c3f3eba-e695-11ee-95c9-87774f69a715 2024-03-20T08:40:07.457 INFO:teuthology.orchestra.run.smithi183.stdout:/usr/bin/ceph: stdout mon_host = [v2:172.21.15.183:3300,v1:172.21.15.183:6789] 2024-03-20T08:40:07.457 INFO:teuthology.orchestra.run.smithi183.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-03-20T08:40:07.457 INFO:teuthology.orchestra.run.smithi183.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-03-20T08:40:07.457 INFO:teuthology.orchestra.run.smithi183.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-03-20T08:40:07.458 INFO:teuthology.orchestra.run.smithi183.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-03-20T08:40:07.458 INFO:teuthology.orchestra.run.smithi183.stdout:/usr/bin/ceph: stdout 2024-03-20T08:40:07.458 INFO:teuthology.orchestra.run.smithi183.stdout:/usr/bin/ceph: stdout [mgr] 2024-03-20T08:40:07.458 INFO:teuthology.orchestra.run.smithi183.stdout:/usr/bin/ceph: stdout mgr/cephadm/use_agent = True 2024-03-20T08:40:07.458 INFO:teuthology.orchestra.run.smithi183.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-03-20T08:40:07.458 INFO:teuthology.orchestra.run.smithi183.stdout:/usr/bin/ceph: stdout 2024-03-20T08:40:07.458 INFO:teuthology.orchestra.run.smithi183.stdout:/usr/bin/ceph: stdout [osd] 2024-03-20T08:40:07.458 INFO:teuthology.orchestra.run.smithi183.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-03-20T08:40:07.458 INFO:teuthology.orchestra.run.smithi183.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-03-20T08:40:07.458 INFO:teuthology.orchestra.run.smithi183.stdout:Generating new minimal ceph.conf... 2024-03-20T08:40:07.578 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:40:07 smithi183 ceph-mon[24901]: from='client.? 172.21.15.183:0/1802948613' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-03-20T08:40:07.578 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:40:07 smithi183 ceph-mon[24901]: from='client.? 172.21.15.183:0/1802948613' entity='client.admin' cmd='[{"prefix": "config assimilate-conf"}]': finished 2024-03-20T08:40:08.478 INFO:teuthology.orchestra.run.smithi183.stdout:Restarting the monitor... 2024-03-20T08:40:08.626 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:40:08 smithi183 ceph-mon[24901]: from='client.? 172.21.15.183:0/2912727759' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-20T08:40:08.626 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:40:08 smithi183 systemd[1]: Stopping Ceph mon.a for 4c3f3eba-e695-11ee-95c9-87774f69a715... 2024-03-20T08:40:08.914 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:40:08 smithi183 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-mon-a[24897]: 2024-03-20T08:40:08.720+0000 7f95343f3700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-03-20T08:40:08.915 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:40:08 smithi183 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-mon-a[24897]: 2024-03-20T08:40:08.720+0000 7f95343f3700 -1 mon.a@0(leader) e1 *** Got Signal Terminated *** 2024-03-20T08:40:09.179 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:40:08 smithi183 podman[25159]: 2024-03-20 08:40:08.915342996 +0000 UTC m=+0.276198577 container died 1e0acaacccdaf0316251161831509e42513469c5ae9fb6f526ebb7dc684b7d20 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae, name=ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-mon-a, GIT_CLEAN=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-yuri5-testing-2024-03-18-1144-e7e80a6, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.build-date=20240307) 2024-03-20T08:40:09.454 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:40:09 smithi183 podman[25159]: 2024-03-20 08:40:09.270911004 +0000 UTC m=+0.631766584 container cleanup 1e0acaacccdaf0316251161831509e42513469c5ae9fb6f526ebb7dc684b7d20 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae, name=ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-mon-a, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, RELEASE=wip-yuri5-testing-2024-03-18-1144-e7e80a6, io.buildah.version=1.33.5, org.label-schema.build-date=20240307) 2024-03-20T08:40:09.454 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:40:09 smithi183 bash[25159]: ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-mon-a 2024-03-20T08:40:09.888 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:40:09 smithi183 podman[25173]: 2024-03-20 08:40:09.546912044 +0000 UTC m=+0.633982564 container remove 1e0acaacccdaf0316251161831509e42513469c5ae9fb6f526ebb7dc684b7d20 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae, name=ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-mon-a, org.label-schema.build-date=20240307, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, CEPH_POINT_RELEASE=, RELEASE=wip-yuri5-testing-2024-03-18-1144-e7e80a6, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0) 2024-03-20T08:40:09.889 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:40:09 smithi183 systemd[1]: ceph-4c3f3eba-e695-11ee-95c9-87774f69a715@mon.a.service: Deactivated successfully. 2024-03-20T08:40:09.889 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:40:09 smithi183 systemd[1]: Stopped Ceph mon.a for 4c3f3eba-e695-11ee-95c9-87774f69a715. 2024-03-20T08:40:09.889 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:40:09 smithi183 systemd[1]: Starting Ceph mon.a for 4c3f3eba-e695-11ee-95c9-87774f69a715... 2024-03-20T08:40:10.144 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:40:09 smithi183 podman[25266]: 2024-03-20 08:40:09.88931887 +0000 UTC m=+0.109245435 container create 8c5d9ae3ddbe9fc75c7c8dd9549797b4bd09ef885926e3aacbd48b29041f5ee2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae, name=ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-mon-a, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240307, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , RELEASE=wip-yuri5-testing-2024-03-18-1144-e7e80a6, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2) 2024-03-20T08:40:10.144 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:40:09 smithi183 podman[25266]: 2024-03-20 08:40:09.798451376 +0000 UTC m=+0.018377899 image pull 9ceba81fb2c8d8682cb159527017d591637b153b3aa4a390ffa8586db6dcda00 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae 2024-03-20T08:40:10.144 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:40:10 smithi183 podman[25266]: 2024-03-20 08:40:10.117743782 +0000 UTC m=+0.337670324 container init 8c5d9ae3ddbe9fc75c7c8dd9549797b4bd09ef885926e3aacbd48b29041f5ee2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae, name=ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-mon-a, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, ceph=True, org.label-schema.build-date=20240307, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, RELEASE=wip-yuri5-testing-2024-03-18-1144-e7e80a6) 2024-03-20T08:40:10.144 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:40:10 smithi183 podman[25266]: 2024-03-20 08:40:10.121048236 +0000 UTC m=+0.340974759 container start 8c5d9ae3ddbe9fc75c7c8dd9549797b4bd09ef885926e3aacbd48b29041f5ee2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae, name=ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-mon-a, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.build-date=20240307, RELEASE=wip-yuri5-testing-2024-03-18-1144-e7e80a6, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0) 2024-03-20T08:40:10.144 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:40:10 smithi183 ceph-mon[25285]: set uid:gid to 167:167 (ceph:ceph) 2024-03-20T08:40:10.144 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:40:10 smithi183 ceph-mon[25285]: ceph version 19.0.0-2153-ge7e80a6a (e7e80a6ac9849f0f23432d0d9832f096730308ae) squid (dev), process ceph-mon, pid 2 2024-03-20T08:40:10.144 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:40:10 smithi183 ceph-mon[25285]: pidfile_write: ignore empty --pid-file 2024-03-20T08:40:10.195 INFO:teuthology.orchestra.run.smithi183.stdout:Setting public_network to 172.21.0.0/20 in mon config section 2024-03-20T08:40:10.428 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:40:10 smithi183 ceph-mon[25285]: load: jerasure load: lrc 2024-03-20T08:40:10.428 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:40:10 smithi183 ceph-mon[25285]: rocksdb: RocksDB version: 7.9.2 2024-03-20T08:40:10.428 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:40:10 smithi183 ceph-mon[25285]: rocksdb: Git sha 0 2024-03-20T08:40:10.428 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:40:10 smithi183 ceph-mon[25285]: rocksdb: Compile date 2024-03-18 19:18:37 2024-03-20T08:40:10.428 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:40:10 smithi183 ceph-mon[25285]: rocksdb: DB SUMMARY 2024-03-20T08:40:10.428 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:40:10 smithi183 ceph-mon[25285]: rocksdb: DB Session ID: 8XCI9U8LLIHO7TP0BDSW 2024-03-20T08:40:10.428 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:40:10 smithi183 ceph-mon[25285]: rocksdb: CURRENT file: CURRENT 2024-03-20T08:40:10.428 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:40:10 smithi183 ceph-mon[25285]: rocksdb: IDENTITY file: IDENTITY 2024-03-20T08:40:10.429 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:40:10 smithi183 ceph-mon[25285]: rocksdb: MANIFEST file: MANIFEST-000010 size: 179 Bytes 2024-03-20T08:40:10.429 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:40:10 smithi183 ceph-mon[25285]: rocksdb: SST files in /var/lib/ceph/mon/ceph-a/store.db dir, Total Num: 1, files: 000008.sst 2024-03-20T08:40:10.429 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:40:10 smithi183 ceph-mon[25285]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-a/store.db: 000009.log size: 85286 ; 2024-03-20T08:40:10.429 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:40:10 smithi183 ceph-mon[25285]: rocksdb: Options.error_if_exists: 0 2024-03-20T08:40:10.429 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:40:10 smithi183 ceph-mon[25285]: rocksdb: Options.create_if_missing: 0 2024-03-20T08:40:10.429 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:40:10 smithi183 ceph-mon[25285]: rocksdb: Options.paranoid_checks: 1 2024-03-20T08:40:10.429 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:40:10 smithi183 ceph-mon[25285]: rocksdb: Options.flush_verify_memtable_count: 1 2024-03-20T08:40:10.429 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:40:10 smithi183 ceph-mon[25285]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-03-20T08:40:10.429 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:40:10 smithi183 ceph-mon[25285]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-03-20T08:40:10.429 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:40:10 smithi183 ceph-mon[25285]: rocksdb: Options.env: 0x556edd17fb20 2024-03-20T08:40:10.429 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:40:10 smithi183 ceph-mon[25285]: rocksdb: Options.fs: PosixFileSystem 2024-03-20T08:40:10.429 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:40:10 smithi183 ceph-mon[25285]: rocksdb: Options.info_log: 0x556ede0cf920 2024-03-20T08:40:10.429 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:40:10 smithi183 ceph-mon[25285]: rocksdb: Options.max_file_opening_threads: 16 2024-03-20T08:40:10.429 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:40:10 smithi183 ceph-mon[25285]: rocksdb: Options.statistics: (nil) 2024-03-20T08:40:10.430 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:40:10 smithi183 ceph-mon[25285]: rocksdb: Options.use_fsync: 0 2024-03-20T08:40:10.430 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:40:10 smithi183 ceph-mon[25285]: rocksdb: Options.max_log_file_size: 0 2024-03-20T08:40:10.430 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:40:10 smithi183 ceph-mon[25285]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-03-20T08:40:10.430 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:40:10 smithi183 ceph-mon[25285]: rocksdb: Options.log_file_time_to_roll: 0 2024-03-20T08:40:10.430 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:40:10 smithi183 ceph-mon[25285]: rocksdb: Options.keep_log_file_num: 1000 2024-03-20T08:40:10.430 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:40:10 smithi183 ceph-mon[25285]: rocksdb: Options.recycle_log_file_num: 0 2024-03-20T08:40:10.430 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:40:10 smithi183 ceph-mon[25285]: rocksdb: Options.allow_fallocate: 1 2024-03-20T08:40:10.430 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:40:10 smithi183 ceph-mon[25285]: rocksdb: Options.allow_mmap_reads: 0 2024-03-20T08:40:10.430 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:40:10 smithi183 ceph-mon[25285]: rocksdb: Options.allow_mmap_writes: 0 2024-03-20T08:40:10.430 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:40:10 smithi183 ceph-mon[25285]: rocksdb: Options.use_direct_reads: 0 2024-03-20T08:40:10.430 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:40:10 smithi183 ceph-mon[25285]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-03-20T08:40:10.430 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:40:10 smithi183 ceph-mon[25285]: rocksdb: Options.create_missing_column_families: 0 2024-03-20T08:40:10.430 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:40:10 smithi183 ceph-mon[25285]: rocksdb: Options.db_log_dir: 2024-03-20T08:40:10.431 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:40:10 smithi183 ceph-mon[25285]: rocksdb: Options.wal_dir: 2024-03-20T08:40:10.431 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:40:10 smithi183 ceph-mon[25285]: rocksdb: Options.table_cache_numshardbits: 6 2024-03-20T08:40:10.431 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:40:10 smithi183 ceph-mon[25285]: rocksdb: Options.WAL_ttl_seconds: 0 2024-03-20T08:40:10.431 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:40:10 smithi183 ceph-mon[25285]: rocksdb: Options.WAL_size_limit_MB: 0 2024-03-20T08:40:10.431 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:40:10 smithi183 ceph-mon[25285]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-03-20T08:40:10.431 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:40:10 smithi183 ceph-mon[25285]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-03-20T08:40:10.431 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:40:10 smithi183 ceph-mon[25285]: rocksdb: Options.is_fd_close_on_exec: 1 2024-03-20T08:40:10.431 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:40:10 smithi183 ceph-mon[25285]: rocksdb: Options.advise_random_on_open: 1 2024-03-20T08:40:10.431 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:40:10 smithi183 ceph-mon[25285]: rocksdb: Options.db_write_buffer_size: 0 2024-03-20T08:40:10.431 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:40:10 smithi183 ceph-mon[25285]: rocksdb: Options.write_buffer_manager: 0x556ede0a21e0 2024-03-20T08:40:10.431 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:40:10 smithi183 ceph-mon[25285]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-03-20T08:40:10.431 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:40:10 smithi183 ceph-mon[25285]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-03-20T08:40:10.431 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:40:10 smithi183 ceph-mon[25285]: rocksdb: Options.use_adaptive_mutex: 0 2024-03-20T08:40:10.432 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:40:10 smithi183 ceph-mon[25285]: rocksdb: Options.rate_limiter: (nil) 2024-03-20T08:40:10.432 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:40:10 smithi183 ceph-mon[25285]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-03-20T08:40:10.432 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:40:10 smithi183 ceph-mon[25285]: rocksdb: Options.wal_recovery_mode: 2 2024-03-20T08:40:10.432 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:40:10 smithi183 ceph-mon[25285]: rocksdb: Options.enable_thread_tracking: 0 2024-03-20T08:40:10.432 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:40:10 smithi183 ceph-mon[25285]: rocksdb: Options.enable_pipelined_write: 0 2024-03-20T08:40:10.432 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:40:10 smithi183 ceph-mon[25285]: rocksdb: Options.unordered_write: 0 2024-03-20T08:40:10.432 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:40:10 smithi183 ceph-mon[25285]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-03-20T08:40:10.432 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:40:10 smithi183 ceph-mon[25285]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-03-20T08:40:10.432 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:40:10 smithi183 ceph-mon[25285]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-03-20T08:40:10.432 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:40:10 smithi183 ceph-mon[25285]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-03-20T08:40:10.432 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:40:10 smithi183 ceph-mon[25285]: rocksdb: Options.row_cache: None 2024-03-20T08:40:10.432 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:40:10 smithi183 ceph-mon[25285]: rocksdb: Options.wal_filter: None 2024-03-20T08:40:10.432 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:40:10 smithi183 ceph-mon[25285]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-03-20T08:40:10.432 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:40:10 smithi183 ceph-mon[25285]: rocksdb: Options.allow_ingest_behind: 0 2024-03-20T08:40:10.433 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:40:10 smithi183 ceph-mon[25285]: rocksdb: Options.two_write_queues: 0 2024-03-20T08:40:10.433 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:40:10 smithi183 ceph-mon[25285]: rocksdb: Options.manual_wal_flush: 0 2024-03-20T08:40:10.433 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:40:10 smithi183 ceph-mon[25285]: rocksdb: Options.wal_compression: 0 2024-03-20T08:40:10.433 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:40:10 smithi183 ceph-mon[25285]: rocksdb: Options.atomic_flush: 0 2024-03-20T08:40:10.433 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:40:10 smithi183 ceph-mon[25285]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-03-20T08:40:10.433 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:40:10 smithi183 ceph-mon[25285]: rocksdb: Options.persist_stats_to_disk: 0 2024-03-20T08:40:10.433 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:40:10 smithi183 ceph-mon[25285]: rocksdb: Options.write_dbid_to_manifest: 0 2024-03-20T08:40:10.433 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:40:10 smithi183 ceph-mon[25285]: rocksdb: Options.log_readahead_size: 0 2024-03-20T08:40:10.433 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:40:10 smithi183 ceph-mon[25285]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-03-20T08:40:10.433 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:40:10 smithi183 ceph-mon[25285]: rocksdb: Options.best_efforts_recovery: 0 2024-03-20T08:40:10.433 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:40:10 smithi183 ceph-mon[25285]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-03-20T08:40:10.433 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:40:10 smithi183 ceph-mon[25285]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-03-20T08:40:10.433 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:40:10 smithi183 ceph-mon[25285]: rocksdb: Options.allow_data_in_errors: 0 2024-03-20T08:40:10.433 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:40:10 smithi183 ceph-mon[25285]: rocksdb: Options.db_host_id: __hostname__ 2024-03-20T08:40:10.433 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:40:10 smithi183 ceph-mon[25285]: rocksdb: Options.enforce_single_del_contracts: true 2024-03-20T08:40:10.434 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:40:10 smithi183 ceph-mon[25285]: rocksdb: Options.max_background_jobs: 2 2024-03-20T08:40:10.434 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:40:10 smithi183 ceph-mon[25285]: rocksdb: Options.max_background_compactions: -1 2024-03-20T08:40:10.434 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:40:10 smithi183 ceph-mon[25285]: rocksdb: Options.max_subcompactions: 1 2024-03-20T08:40:10.435 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:40:10 smithi183 ceph-mon[25285]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-03-20T08:40:10.435 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:40:10 smithi183 ceph-mon[25285]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-03-20T08:40:10.435 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:40:10 smithi183 ceph-mon[25285]: rocksdb: Options.delayed_write_rate : 16777216 2024-03-20T08:40:10.435 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:40:10 smithi183 ceph-mon[25285]: rocksdb: Options.max_total_wal_size: 0 2024-03-20T08:40:10.436 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:40:10 smithi183 ceph-mon[25285]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-03-20T08:40:10.436 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:40:10 smithi183 ceph-mon[25285]: rocksdb: Options.stats_dump_period_sec: 600 2024-03-20T08:40:10.436 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:40:10 smithi183 ceph-mon[25285]: rocksdb: Options.stats_persist_period_sec: 600 2024-03-20T08:40:10.436 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:40:10 smithi183 ceph-mon[25285]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-03-20T08:40:10.436 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:40:10 smithi183 ceph-mon[25285]: rocksdb: Options.max_open_files: -1 2024-03-20T08:40:10.436 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:40:10 smithi183 ceph-mon[25285]: rocksdb: Options.bytes_per_sync: 0 2024-03-20T08:40:10.436 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:40:10 smithi183 ceph-mon[25285]: rocksdb: Options.wal_bytes_per_sync: 0 2024-03-20T08:40:10.436 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:40:10 smithi183 ceph-mon[25285]: rocksdb: Options.strict_bytes_per_sync: 0 2024-03-20T08:40:10.436 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:40:10 smithi183 ceph-mon[25285]: rocksdb: Options.compaction_readahead_size: 0 2024-03-20T08:40:10.436 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:40:10 smithi183 ceph-mon[25285]: rocksdb: Options.max_background_flushes: -1 2024-03-20T08:40:10.436 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:40:10 smithi183 ceph-mon[25285]: rocksdb: Compression algorithms supported: 2024-03-20T08:40:10.436 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:40:10 smithi183 ceph-mon[25285]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-03-20T08:40:10.436 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:40:10 smithi183 ceph-mon[25285]: rocksdb: kZSTD supported: 0 2024-03-20T08:40:10.436 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:40:10 smithi183 ceph-mon[25285]: rocksdb: kXpressCompression supported: 0 2024-03-20T08:40:10.437 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:40:10 smithi183 ceph-mon[25285]: rocksdb: kLZ4HCCompression supported: 1 2024-03-20T08:40:10.437 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:40:10 smithi183 ceph-mon[25285]: rocksdb: kZlibCompression supported: 1 2024-03-20T08:40:10.437 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:40:10 smithi183 ceph-mon[25285]: rocksdb: kSnappyCompression supported: 1 2024-03-20T08:40:10.437 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:40:10 smithi183 ceph-mon[25285]: rocksdb: kLZ4Compression supported: 1 2024-03-20T08:40:10.437 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:40:10 smithi183 ceph-mon[25285]: rocksdb: kBZip2Compression supported: 0 2024-03-20T08:40:10.437 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:40:10 smithi183 ceph-mon[25285]: rocksdb: Fast CRC32 supported: Supported on x86 2024-03-20T08:40:10.437 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:40:10 smithi183 ceph-mon[25285]: rocksdb: DMutex implementation: pthread_mutex_t 2024-03-20T08:40:10.437 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:40:10 smithi183 ceph-mon[25285]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000010 2024-03-20T08:40:10.437 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:40:10 smithi183 ceph-mon[25285]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-03-20T08:40:10.437 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:40:10 smithi183 ceph-mon[25285]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-03-20T08:40:10.437 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:40:10 smithi183 ceph-mon[25285]: rocksdb: Options.merge_operator: 2024-03-20T08:40:10.437 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:40:10 smithi183 ceph-mon[25285]: rocksdb: Options.compaction_filter: None 2024-03-20T08:40:10.437 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:40:10 smithi183 ceph-mon[25285]: rocksdb: Options.compaction_filter_factory: None 2024-03-20T08:40:10.437 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:40:10 smithi183 ceph-mon[25285]: rocksdb: Options.sst_partitioner_factory: None 2024-03-20T08:40:10.438 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:40:10 smithi183 ceph-mon[25285]: rocksdb: Options.memtable_factory: SkipListFactory 2024-03-20T08:40:10.438 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:40:10 smithi183 ceph-mon[25285]: rocksdb: Options.table_factory: BlockBasedTable 2024-03-20T08:40:10.438 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:40:10 smithi183 ceph-mon[25285]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x556ede0cfa20) 2024-03-20T08:40:10.438 INFO:journalctl@ceph.mon.a.smithi183.stdout: cache_index_and_filter_blocks: 1 2024-03-20T08:40:10.438 INFO:journalctl@ceph.mon.a.smithi183.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-03-20T08:40:10.438 INFO:journalctl@ceph.mon.a.smithi183.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-03-20T08:40:10.438 INFO:journalctl@ceph.mon.a.smithi183.stdout: pin_top_level_index_and_filter: 1 2024-03-20T08:40:10.438 INFO:journalctl@ceph.mon.a.smithi183.stdout: index_type: 0 2024-03-20T08:40:10.438 INFO:journalctl@ceph.mon.a.smithi183.stdout: data_block_index_type: 0 2024-03-20T08:40:10.438 INFO:journalctl@ceph.mon.a.smithi183.stdout: index_shortening: 1 2024-03-20T08:40:10.438 INFO:journalctl@ceph.mon.a.smithi183.stdout: data_block_hash_table_util_ratio: 0.750000 2024-03-20T08:40:10.438 INFO:journalctl@ceph.mon.a.smithi183.stdout: checksum: 4 2024-03-20T08:40:10.438 INFO:journalctl@ceph.mon.a.smithi183.stdout: no_block_cache: 0 2024-03-20T08:40:10.439 INFO:journalctl@ceph.mon.a.smithi183.stdout: block_cache: 0x556ede0d5090 2024-03-20T08:40:10.439 INFO:journalctl@ceph.mon.a.smithi183.stdout: block_cache_name: BinnedLRUCache 2024-03-20T08:40:10.439 INFO:journalctl@ceph.mon.a.smithi183.stdout: block_cache_options: 2024-03-20T08:40:10.439 INFO:journalctl@ceph.mon.a.smithi183.stdout: capacity : 536870912 2024-03-20T08:40:10.439 INFO:journalctl@ceph.mon.a.smithi183.stdout: num_shard_bits : 4 2024-03-20T08:40:10.439 INFO:journalctl@ceph.mon.a.smithi183.stdout: strict_capacity_limit : 0 2024-03-20T08:40:10.439 INFO:journalctl@ceph.mon.a.smithi183.stdout: high_pri_pool_ratio: 0.000 2024-03-20T08:40:10.439 INFO:journalctl@ceph.mon.a.smithi183.stdout: block_cache_compressed: (nil) 2024-03-20T08:40:10.439 INFO:journalctl@ceph.mon.a.smithi183.stdout: persistent_cache: (nil) 2024-03-20T08:40:10.439 INFO:journalctl@ceph.mon.a.smithi183.stdout: block_size: 4096 2024-03-20T08:40:10.439 INFO:journalctl@ceph.mon.a.smithi183.stdout: block_size_deviation: 10 2024-03-20T08:40:10.439 INFO:journalctl@ceph.mon.a.smithi183.stdout: block_restart_interval: 16 2024-03-20T08:40:10.439 INFO:journalctl@ceph.mon.a.smithi183.stdout: index_block_restart_interval: 1 2024-03-20T08:40:10.439 INFO:journalctl@ceph.mon.a.smithi183.stdout: metadata_block_size: 4096 2024-03-20T08:40:10.439 INFO:journalctl@ceph.mon.a.smithi183.stdout: partition_filters: 0 2024-03-20T08:40:10.440 INFO:journalctl@ceph.mon.a.smithi183.stdout: use_delta_encoding: 1 2024-03-20T08:40:10.440 INFO:journalctl@ceph.mon.a.smithi183.stdout: filter_policy: bloomfilter 2024-03-20T08:40:10.440 INFO:journalctl@ceph.mon.a.smithi183.stdout: whole_key_filtering: 1 2024-03-20T08:40:10.440 INFO:journalctl@ceph.mon.a.smithi183.stdout: verify_compression: 0 2024-03-20T08:40:10.440 INFO:journalctl@ceph.mon.a.smithi183.stdout: read_amp_bytes_per_bit: 0 2024-03-20T08:40:10.440 INFO:journalctl@ceph.mon.a.smithi183.stdout: format_version: 5 2024-03-20T08:40:10.440 INFO:journalctl@ceph.mon.a.smithi183.stdout: enable_index_compression: 1 2024-03-20T08:40:10.440 INFO:journalctl@ceph.mon.a.smithi183.stdout: block_align: 0 2024-03-20T08:40:10.440 INFO:journalctl@ceph.mon.a.smithi183.stdout: max_auto_readahead_size: 262144 2024-03-20T08:40:10.440 INFO:journalctl@ceph.mon.a.smithi183.stdout: prepopulate_block_cache: 0 2024-03-20T08:40:10.440 INFO:journalctl@ceph.mon.a.smithi183.stdout: initial_auto_readahead_size: 8192 2024-03-20T08:40:10.440 INFO:journalctl@ceph.mon.a.smithi183.stdout: num_file_reads_for_auto_readahead: 2 2024-03-20T08:40:10.440 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:40:10 smithi183 ceph-mon[25285]: rocksdb: Options.write_buffer_size: 33554432 2024-03-20T08:40:10.440 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:40:10 smithi183 ceph-mon[25285]: rocksdb: Options.max_write_buffer_number: 2 2024-03-20T08:40:10.440 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:40:10 smithi183 ceph-mon[25285]: rocksdb: Options.compression: NoCompression 2024-03-20T08:40:10.441 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:40:10 smithi183 ceph-mon[25285]: rocksdb: Options.bottommost_compression: Disabled 2024-03-20T08:40:10.441 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:40:10 smithi183 ceph-mon[25285]: rocksdb: Options.prefix_extractor: nullptr 2024-03-20T08:40:10.441 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:40:10 smithi183 ceph-mon[25285]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-03-20T08:40:10.441 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:40:10 smithi183 ceph-mon[25285]: rocksdb: Options.num_levels: 7 2024-03-20T08:40:10.441 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:40:10 smithi183 ceph-mon[25285]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-03-20T08:40:10.441 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:40:10 smithi183 ceph-mon[25285]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-03-20T08:40:10.441 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:40:10 smithi183 ceph-mon[25285]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-03-20T08:40:10.441 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:40:10 smithi183 ceph-mon[25285]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-03-20T08:40:10.441 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:40:10 smithi183 ceph-mon[25285]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-03-20T08:40:10.441 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:40:10 smithi183 ceph-mon[25285]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-03-20T08:40:10.441 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:40:10 smithi183 ceph-mon[25285]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-03-20T08:40:10.441 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:40:10 smithi183 ceph-mon[25285]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-03-20T08:40:10.441 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:40:10 smithi183 ceph-mon[25285]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-03-20T08:40:10.441 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:40:10 smithi183 ceph-mon[25285]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-03-20T08:40:10.442 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:40:10 smithi183 ceph-mon[25285]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-03-20T08:40:10.442 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:40:10 smithi183 ceph-mon[25285]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-03-20T08:40:10.442 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:40:10 smithi183 ceph-mon[25285]: rocksdb: Options.compression_opts.window_bits: -14 2024-03-20T08:40:10.442 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:40:10 smithi183 ceph-mon[25285]: rocksdb: Options.compression_opts.level: 32767 2024-03-20T08:40:10.442 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:40:10 smithi183 ceph-mon[25285]: rocksdb: Options.compression_opts.strategy: 0 2024-03-20T08:40:10.442 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:40:10 smithi183 ceph-mon[25285]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-03-20T08:40:10.442 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:40:10 smithi183 ceph-mon[25285]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-03-20T08:40:10.442 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:40:10 smithi183 ceph-mon[25285]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-03-20T08:40:10.442 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:40:10 smithi183 ceph-mon[25285]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-03-20T08:40:10.442 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:40:10 smithi183 ceph-mon[25285]: rocksdb: Options.compression_opts.enabled: false 2024-03-20T08:40:10.442 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:40:10 smithi183 ceph-mon[25285]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-03-20T08:40:10.442 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:40:10 smithi183 ceph-mon[25285]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-03-20T08:40:10.442 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:40:10 smithi183 ceph-mon[25285]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-03-20T08:40:10.442 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:40:10 smithi183 ceph-mon[25285]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-03-20T08:40:10.443 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:40:10 smithi183 ceph-mon[25285]: rocksdb: Options.target_file_size_base: 67108864 2024-03-20T08:40:10.443 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:40:10 smithi183 ceph-mon[25285]: rocksdb: Options.target_file_size_multiplier: 1 2024-03-20T08:40:10.443 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:40:10 smithi183 ceph-mon[25285]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-03-20T08:40:10.443 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:40:10 smithi183 ceph-mon[25285]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-03-20T08:40:10.443 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:40:10 smithi183 ceph-mon[25285]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-03-20T08:40:10.443 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:40:10 smithi183 ceph-mon[25285]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-03-20T08:40:10.443 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:40:10 smithi183 ceph-mon[25285]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-03-20T08:40:10.443 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:40:10 smithi183 ceph-mon[25285]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-03-20T08:40:10.443 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:40:10 smithi183 ceph-mon[25285]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-03-20T08:40:10.443 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:40:10 smithi183 ceph-mon[25285]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-03-20T08:40:10.443 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:40:10 smithi183 ceph-mon[25285]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-03-20T08:40:10.443 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:40:10 smithi183 ceph-mon[25285]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-03-20T08:40:10.443 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:40:10 smithi183 ceph-mon[25285]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-03-20T08:40:10.443 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:40:10 smithi183 ceph-mon[25285]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-03-20T08:40:10.444 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:40:10 smithi183 ceph-mon[25285]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-03-20T08:40:10.444 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:40:10 smithi183 ceph-mon[25285]: rocksdb: Options.arena_block_size: 1048576 2024-03-20T08:40:10.444 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:40:10 smithi183 ceph-mon[25285]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-03-20T08:40:10.444 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:40:10 smithi183 ceph-mon[25285]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-03-20T08:40:10.444 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:40:10 smithi183 ceph-mon[25285]: rocksdb: Options.disable_auto_compactions: 0 2024-03-20T08:40:10.444 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:40:10 smithi183 ceph-mon[25285]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-03-20T08:40:10.444 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:40:10 smithi183 ceph-mon[25285]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-03-20T08:40:10.444 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:40:10 smithi183 ceph-mon[25285]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-03-20T08:40:10.444 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:40:10 smithi183 ceph-mon[25285]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-03-20T08:40:10.444 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:40:10 smithi183 ceph-mon[25285]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-03-20T08:40:10.444 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:40:10 smithi183 ceph-mon[25285]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-03-20T08:40:10.444 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:40:10 smithi183 ceph-mon[25285]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-03-20T08:40:10.444 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:40:10 smithi183 ceph-mon[25285]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-03-20T08:40:10.444 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:40:10 smithi183 ceph-mon[25285]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-03-20T08:40:10.444 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:40:10 smithi183 ceph-mon[25285]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-03-20T08:40:10.445 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:40:10 smithi183 ceph-mon[25285]: rocksdb: Options.table_properties_collectors: 2024-03-20T08:40:10.445 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:40:10 smithi183 ceph-mon[25285]: rocksdb: Options.inplace_update_support: 0 2024-03-20T08:40:10.445 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:40:10 smithi183 ceph-mon[25285]: rocksdb: Options.inplace_update_num_locks: 10000 2024-03-20T08:40:10.445 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:40:10 smithi183 ceph-mon[25285]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-03-20T08:40:10.445 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:40:10 smithi183 ceph-mon[25285]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-03-20T08:40:10.445 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:40:10 smithi183 ceph-mon[25285]: rocksdb: Options.memtable_huge_page_size: 0 2024-03-20T08:40:10.445 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:40:10 smithi183 ceph-mon[25285]: rocksdb: Options.bloom_locality: 0 2024-03-20T08:40:10.445 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:40:10 smithi183 ceph-mon[25285]: rocksdb: Options.max_successive_merges: 0 2024-03-20T08:40:10.445 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:40:10 smithi183 ceph-mon[25285]: rocksdb: Options.optimize_filters_for_hits: 0 2024-03-20T08:40:10.445 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:40:10 smithi183 ceph-mon[25285]: rocksdb: Options.paranoid_file_checks: 0 2024-03-20T08:40:10.445 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:40:10 smithi183 ceph-mon[25285]: rocksdb: Options.force_consistency_checks: 1 2024-03-20T08:40:10.445 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:40:10 smithi183 ceph-mon[25285]: rocksdb: Options.report_bg_io_stats: 0 2024-03-20T08:40:10.445 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:40:10 smithi183 ceph-mon[25285]: rocksdb: Options.ttl: 2592000 2024-03-20T08:40:10.445 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:40:10 smithi183 ceph-mon[25285]: rocksdb: Options.periodic_compaction_seconds: 0 2024-03-20T08:40:10.446 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:40:10 smithi183 ceph-mon[25285]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-03-20T08:40:10.446 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:40:10 smithi183 ceph-mon[25285]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-03-20T08:40:10.446 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:40:10 smithi183 ceph-mon[25285]: rocksdb: Options.enable_blob_files: false 2024-03-20T08:40:10.446 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:40:10 smithi183 ceph-mon[25285]: rocksdb: Options.min_blob_size: 0 2024-03-20T08:40:10.446 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:40:10 smithi183 ceph-mon[25285]: rocksdb: Options.blob_file_size: 268435456 2024-03-20T08:40:10.446 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:40:10 smithi183 ceph-mon[25285]: rocksdb: Options.blob_compression_type: NoCompression 2024-03-20T08:40:10.446 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:40:10 smithi183 ceph-mon[25285]: rocksdb: Options.enable_blob_garbage_collection: false 2024-03-20T08:40:10.446 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:40:10 smithi183 ceph-mon[25285]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-03-20T08:40:10.446 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:40:10 smithi183 ceph-mon[25285]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-03-20T08:40:10.446 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:40:10 smithi183 ceph-mon[25285]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-03-20T08:40:10.447 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:40:10 smithi183 ceph-mon[25285]: rocksdb: Options.blob_file_starting_level: 0 2024-03-20T08:40:10.447 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:40:10 smithi183 ceph-mon[25285]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-03-20T08:40:10.447 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:40:10 smithi183 ceph-mon[25285]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000010 succeeded,manifest_file_number is 10, next_file_number is 12, last_sequence is 5, log_number is 5,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 5 2024-03-20T08:40:10.447 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:40:10 smithi183 ceph-mon[25285]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 5 2024-03-20T08:40:10.447 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:40:10 smithi183 ceph-mon[25285]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: 4977e935-c96c-4c07-b6f2-a0f0cc9c2db1 2024-03-20T08:40:10.447 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:40:10 smithi183 ceph-mon[25285]: rocksdb: EVENT_LOG_v1 {"time_micros": 1710924010150497, "job": 1, "event": "recovery_started", "wal_files": [9]} 2024-03-20T08:40:10.447 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:40:10 smithi183 ceph-mon[25285]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #9 mode 2 2024-03-20T08:40:10.447 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:40:10 smithi183 ceph-mon[25285]: rocksdb: EVENT_LOG_v1 {"time_micros": 1710924010151335, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 13, "file_size": 81800, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 8, "largest_seqno": 258, "table_properties": {"data_size": 79969, "index_size": 241, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 581, "raw_key_size": 10057, "raw_average_key_size": 47, "raw_value_size": 74233, "raw_average_value_size": 348, "num_data_blocks": 11, "num_entries": 213, "num_filter_entries": 213, "num_deletions": 3, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1710924010, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "4977e935-c96c-4c07-b6f2-a0f0cc9c2db1", "db_session_id": "8XCI9U8LLIHO7TP0BDSW", "orig_file_number": 13, "seqno_to_time_mapping": "N/A"}} 2024-03-20T08:40:10.447 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:40:10 smithi183 ceph-mon[25285]: rocksdb: EVENT_LOG_v1 {"time_micros": 1710924010151414, "job": 1, "event": "recovery_finished"} 2024-03-20T08:40:10.447 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:40:10 smithi183 ceph-mon[25285]: rocksdb: [db/version_set.cc:5047] Creating manifest 15 2024-03-20T08:40:10.447 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:40:10 smithi183 ceph-mon[25285]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-a/store.db/000009.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-03-20T08:40:10.447 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:40:10 smithi183 ceph-mon[25285]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x556ede1c8000 2024-03-20T08:40:10.447 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:40:10 smithi183 ceph-mon[25285]: rocksdb: DB pointer 0x556ede1b2000 2024-03-20T08:40:10.447 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:40:10 smithi183 ceph-mon[25285]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-03-20T08:40:10.448 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:40:10 smithi183 ceph-mon[25285]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-03-20T08:40:10.448 INFO:journalctl@ceph.mon.a.smithi183.stdout: ** DB Stats ** 2024-03-20T08:40:10.448 INFO:journalctl@ceph.mon.a.smithi183.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-03-20T08:40:10.448 INFO:journalctl@ceph.mon.a.smithi183.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-03-20T08:40:10.448 INFO:journalctl@ceph.mon.a.smithi183.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-03-20T08:40:10.448 INFO:journalctl@ceph.mon.a.smithi183.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-03-20T08:40:10.448 INFO:journalctl@ceph.mon.a.smithi183.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-03-20T08:40:10.448 INFO:journalctl@ceph.mon.a.smithi183.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-03-20T08:40:10.448 INFO:journalctl@ceph.mon.a.smithi183.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-03-20T08:40:10.448 INFO:journalctl@ceph.mon.a.smithi183.stdout: 2024-03-20T08:40:10.448 INFO:journalctl@ceph.mon.a.smithi183.stdout: ** Compaction Stats [default] ** 2024-03-20T08:40:10.448 INFO:journalctl@ceph.mon.a.smithi183.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-03-20T08:40:10.448 INFO:journalctl@ceph.mon.a.smithi183.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-03-20T08:40:10.448 INFO:journalctl@ceph.mon.a.smithi183.stdout: L0 2/0 81.72 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 146.4 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-03-20T08:40:10.449 INFO:journalctl@ceph.mon.a.smithi183.stdout: Sum 2/0 81.72 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 146.4 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-03-20T08:40:10.449 INFO:journalctl@ceph.mon.a.smithi183.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 146.4 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-03-20T08:40:10.449 INFO:journalctl@ceph.mon.a.smithi183.stdout: 2024-03-20T08:40:10.449 INFO:journalctl@ceph.mon.a.smithi183.stdout: ** Compaction Stats [default] ** 2024-03-20T08:40:10.449 INFO:journalctl@ceph.mon.a.smithi183.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-03-20T08:40:10.449 INFO:journalctl@ceph.mon.a.smithi183.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-03-20T08:40:10.449 INFO:journalctl@ceph.mon.a.smithi183.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 146.4 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-03-20T08:40:10.449 INFO:journalctl@ceph.mon.a.smithi183.stdout: 2024-03-20T08:40:10.449 INFO:journalctl@ceph.mon.a.smithi183.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-03-20T08:40:10.449 INFO:journalctl@ceph.mon.a.smithi183.stdout: 2024-03-20T08:40:10.449 INFO:journalctl@ceph.mon.a.smithi183.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-03-20T08:40:10.449 INFO:journalctl@ceph.mon.a.smithi183.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-03-20T08:40:10.449 INFO:journalctl@ceph.mon.a.smithi183.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-03-20T08:40:10.449 INFO:journalctl@ceph.mon.a.smithi183.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-03-20T08:40:10.449 INFO:journalctl@ceph.mon.a.smithi183.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-03-20T08:40:10.450 INFO:journalctl@ceph.mon.a.smithi183.stdout: AddFile(Keys): cumulative 0, interval 0 2024-03-20T08:40:10.450 INFO:journalctl@ceph.mon.a.smithi183.stdout: Cumulative compaction: 0.00 GB write, 28.56 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-20T08:40:10.450 INFO:journalctl@ceph.mon.a.smithi183.stdout: Interval compaction: 0.00 GB write, 28.56 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-20T08:40:10.450 INFO:journalctl@ceph.mon.a.smithi183.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-03-20T08:40:10.450 INFO:journalctl@ceph.mon.a.smithi183.stdout: Block cache BinnedLRUCache@0x556ede0d5090#2 capacity: 512.00 MB usage: 1.20 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 8e-06 secs_since: 0 2024-03-20T08:40:10.450 INFO:journalctl@ceph.mon.a.smithi183.stdout: Block cache entry stats(count,size,portion): FilterBlock(2,0.77 KB,0.000146031%) IndexBlock(2,0.44 KB,8.34465e-05%) Misc(1,0.00 KB,0%) 2024-03-20T08:40:10.450 INFO:journalctl@ceph.mon.a.smithi183.stdout: 2024-03-20T08:40:10.450 INFO:journalctl@ceph.mon.a.smithi183.stdout: ** File Read Latency Histogram By Level [default] ** 2024-03-20T08:40:10.450 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:40:10 smithi183 ceph-mon[25285]: starting mon.a rank 0 at public addrs [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] at bind addrs [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] mon_data /var/lib/ceph/mon/ceph-a fsid 4c3f3eba-e695-11ee-95c9-87774f69a715 2024-03-20T08:40:10.450 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:40:10 smithi183 ceph-mon[25285]: mon.a@-1(???) e1 preinit fsid 4c3f3eba-e695-11ee-95c9-87774f69a715 2024-03-20T08:40:10.450 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:40:10 smithi183 ceph-mon[25285]: mon.a@-1(???).mds e1 new map 2024-03-20T08:40:10.450 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:40:10 smithi183 ceph-mon[25285]: mon.a@-1(???).mds e1 print_map 2024-03-20T08:40:10.450 INFO:journalctl@ceph.mon.a.smithi183.stdout: e1 2024-03-20T08:40:10.450 INFO:journalctl@ceph.mon.a.smithi183.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-03-20T08:40:10.451 INFO:journalctl@ceph.mon.a.smithi183.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments} 2024-03-20T08:40:10.451 INFO:journalctl@ceph.mon.a.smithi183.stdout: legacy client fscid: -1 2024-03-20T08:40:10.451 INFO:journalctl@ceph.mon.a.smithi183.stdout: 2024-03-20T08:40:10.451 INFO:journalctl@ceph.mon.a.smithi183.stdout: No filesystems configured 2024-03-20T08:40:10.451 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:40:10 smithi183 ceph-mon[25285]: mon.a@-1(???).osd e1 crush map has features 3314932999778484224, adjusting msgr requires 2024-03-20T08:40:10.451 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:40:10 smithi183 ceph-mon[25285]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-03-20T08:40:10.451 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:40:10 smithi183 ceph-mon[25285]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-03-20T08:40:10.451 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:40:10 smithi183 ceph-mon[25285]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-03-20T08:40:10.451 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:40:10 smithi183 ceph-mon[25285]: mon.a@-1(???).paxosservice(auth 1..2) refresh upgraded, format 0 -> 3 2024-03-20T08:40:10.451 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:40:10 smithi183 bash[25266]: 8c5d9ae3ddbe9fc75c7c8dd9549797b4bd09ef885926e3aacbd48b29041f5ee2 2024-03-20T08:40:10.451 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:40:10 smithi183 systemd[1]: Started Ceph mon.a for 4c3f3eba-e695-11ee-95c9-87774f69a715. 2024-03-20T08:40:10.451 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:40:10 smithi183 ceph-mon[25285]: mon.a is new leader, mons a in quorum (ranks 0) 2024-03-20T08:40:10.451 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:40:10 smithi183 ceph-mon[25285]: monmap e1: 1 mons at {a=[v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-20T08:40:10.451 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:40:10 smithi183 ceph-mon[25285]: fsmap 2024-03-20T08:40:10.452 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:40:10 smithi183 ceph-mon[25285]: osdmap e1: 0 total, 0 up, 0 in 2024-03-20T08:40:10.452 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:40:10 smithi183 ceph-mon[25285]: mgrmap e1: no daemons active 2024-03-20T08:40:11.954 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:40:11 smithi183 ceph-mon[25285]: from='client.? 172.21.15.183:0/76560225' entity='client.admin' 2024-03-20T08:40:12.596 INFO:teuthology.orchestra.run.smithi183.stdout:Wrote config to /etc/ceph/ceph.conf 2024-03-20T08:40:12.617 INFO:teuthology.orchestra.run.smithi183.stdout:Wrote keyring to /etc/ceph/ceph.client.admin.keyring 2024-03-20T08:40:12.617 INFO:teuthology.orchestra.run.smithi183.stdout:Creating mgr... 2024-03-20T08:40:12.617 INFO:teuthology.orchestra.run.smithi183.stdout:Verifying port 0.0.0.0:9283 ... 2024-03-20T08:40:12.618 INFO:teuthology.orchestra.run.smithi183.stdout:Verifying port 0.0.0.0:8765 ... 2024-03-20T08:40:12.866 INFO:teuthology.orchestra.run.smithi183.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-4c3f3eba-e695-11ee-95c9-87774f69a715@mgr.y 2024-03-20T08:40:12.866 INFO:teuthology.orchestra.run.smithi183.stdout:systemctl: stderr Failed to reset failed state of unit ceph-4c3f3eba-e695-11ee-95c9-87774f69a715@mgr.y.service: Unit ceph-4c3f3eba-e695-11ee-95c9-87774f69a715@mgr.y.service not loaded. 2024-03-20T08:40:13.014 INFO:teuthology.orchestra.run.smithi183.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-4c3f3eba-e695-11ee-95c9-87774f69a715.target.wants/ceph-4c3f3eba-e695-11ee-95c9-87774f69a715@mgr.y.service → /etc/systemd/system/ceph-4c3f3eba-e695-11ee-95c9-87774f69a715@.service. 2024-03-20T08:40:13.204 INFO:journalctl@ceph.mgr.y.smithi183.stdout:Mar 20 08:40:13 smithi183 systemd[1]: Starting Ceph mgr.y for 4c3f3eba-e695-11ee-95c9-87774f69a715... 2024-03-20T08:40:13.529 INFO:journalctl@ceph.mgr.y.smithi183.stdout:Mar 20 08:40:13 smithi183 podman[25498]: 2024-03-20 08:40:13.252724787 +0000 UTC m=+0.114487469 container create 33072c5d6210372d85db1be2dd63399a8e19a70ce4c0d2dd734057a08ed26f47 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae, name=ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-mgr-y, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, RELEASE=wip-yuri5-testing-2024-03-18-1144-e7e80a6, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240307, org.label-schema.schema-version=1.0, GIT_CLEAN=True) 2024-03-20T08:40:13.529 INFO:journalctl@ceph.mgr.y.smithi183.stdout:Mar 20 08:40:13 smithi183 podman[25498]: 2024-03-20 08:40:13.155638302 +0000 UTC m=+0.017400987 image pull 9ceba81fb2c8d8682cb159527017d591637b153b3aa4a390ffa8586db6dcda00 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae 2024-03-20T08:40:13.529 INFO:journalctl@ceph.mgr.y.smithi183.stdout:Mar 20 08:40:13 smithi183 podman[25498]: 2024-03-20 08:40:13.471747232 +0000 UTC m=+0.333509913 container init 33072c5d6210372d85db1be2dd63399a8e19a70ce4c0d2dd734057a08ed26f47 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae, name=ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-mgr-y, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, RELEASE=wip-yuri5-testing-2024-03-18-1144-e7e80a6, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240307, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux ) 2024-03-20T08:40:13.529 INFO:journalctl@ceph.mgr.y.smithi183.stdout:Mar 20 08:40:13 smithi183 podman[25498]: 2024-03-20 08:40:13.474278357 +0000 UTC m=+0.336041047 container start 33072c5d6210372d85db1be2dd63399a8e19a70ce4c0d2dd734057a08ed26f47 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae, name=ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-mgr-y, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_CLEAN=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-yuri5-testing-2024-03-18-1144-e7e80a6, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240307, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.vendor=CentOS) 2024-03-20T08:40:13.542 INFO:teuthology.orchestra.run.smithi183.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-03-20T08:40:13.542 INFO:teuthology.orchestra.run.smithi183.stdout:systemctl: stdout disabled 2024-03-20T08:40:13.550 INFO:teuthology.orchestra.run.smithi183.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-03-20T08:40:13.550 INFO:teuthology.orchestra.run.smithi183.stdout:systemctl: stdout inactive 2024-03-20T08:40:13.550 INFO:teuthology.orchestra.run.smithi183.stdout:firewalld.service is not enabled 2024-03-20T08:40:13.550 INFO:teuthology.orchestra.run.smithi183.stdout:Not possible to enable service . firewalld.service is not available 2024-03-20T08:40:13.558 INFO:teuthology.orchestra.run.smithi183.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-03-20T08:40:13.558 INFO:teuthology.orchestra.run.smithi183.stdout:systemctl: stdout disabled 2024-03-20T08:40:13.565 INFO:teuthology.orchestra.run.smithi183.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-03-20T08:40:13.565 INFO:teuthology.orchestra.run.smithi183.stdout:systemctl: stdout inactive 2024-03-20T08:40:13.565 INFO:teuthology.orchestra.run.smithi183.stdout:firewalld.service is not enabled 2024-03-20T08:40:13.565 INFO:teuthology.orchestra.run.smithi183.stdout:Not possible to open ports <[9283, 8765]>. firewalld.service is not available 2024-03-20T08:40:13.565 INFO:teuthology.orchestra.run.smithi183.stdout:Waiting for mgr to start... 2024-03-20T08:40:13.565 INFO:teuthology.orchestra.run.smithi183.stdout:Waiting for mgr... 2024-03-20T08:40:13.781 INFO:journalctl@ceph.mgr.y.smithi183.stdout:Mar 20 08:40:13 smithi183 bash[25498]: 33072c5d6210372d85db1be2dd63399a8e19a70ce4c0d2dd734057a08ed26f47 2024-03-20T08:40:13.781 INFO:journalctl@ceph.mgr.y.smithi183.stdout:Mar 20 08:40:13 smithi183 systemd[1]: Started Ceph mgr.y for 4c3f3eba-e695-11ee-95c9-87774f69a715. 2024-03-20T08:40:14.361 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:40:14 smithi183 ceph-mon[25285]: from='client.? 172.21.15.183:0/3199536440' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-03-20T08:40:14.362 INFO:journalctl@ceph.mgr.y.smithi183.stdout:Mar 20 08:40:14 smithi183 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-mgr-y[25513]: 2024-03-20T08:40:14.035+0000 7f2a95edb200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-20T08:40:14.362 INFO:journalctl@ceph.mgr.y.smithi183.stdout:Mar 20 08:40:14 smithi183 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-mgr-y[25513]: 2024-03-20T08:40:14.138+0000 7f2a95edb200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-20T08:40:14.600 INFO:teuthology.orchestra.run.smithi183.stdout:/usr/bin/ceph: stdout 2024-03-20T08:40:14.600 INFO:teuthology.orchestra.run.smithi183.stdout:/usr/bin/ceph: stdout { 2024-03-20T08:40:14.600 INFO:teuthology.orchestra.run.smithi183.stdout:/usr/bin/ceph: stdout "fsid": "4c3f3eba-e695-11ee-95c9-87774f69a715", 2024-03-20T08:40:14.600 INFO:teuthology.orchestra.run.smithi183.stdout:/usr/bin/ceph: stdout "health": { 2024-03-20T08:40:14.600 INFO:teuthology.orchestra.run.smithi183.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-03-20T08:40:14.600 INFO:teuthology.orchestra.run.smithi183.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-03-20T08:40:14.600 INFO:teuthology.orchestra.run.smithi183.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-03-20T08:40:14.600 INFO:teuthology.orchestra.run.smithi183.stdout:/usr/bin/ceph: stdout }, 2024-03-20T08:40:14.600 INFO:teuthology.orchestra.run.smithi183.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-03-20T08:40:14.601 INFO:teuthology.orchestra.run.smithi183.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-03-20T08:40:14.601 INFO:teuthology.orchestra.run.smithi183.stdout:/usr/bin/ceph: stdout 0 2024-03-20T08:40:14.601 INFO:teuthology.orchestra.run.smithi183.stdout:/usr/bin/ceph: stdout ], 2024-03-20T08:40:14.601 INFO:teuthology.orchestra.run.smithi183.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-03-20T08:40:14.601 INFO:teuthology.orchestra.run.smithi183.stdout:/usr/bin/ceph: stdout "a" 2024-03-20T08:40:14.601 INFO:teuthology.orchestra.run.smithi183.stdout:/usr/bin/ceph: stdout ], 2024-03-20T08:40:14.601 INFO:teuthology.orchestra.run.smithi183.stdout:/usr/bin/ceph: stdout "quorum_age": 3, 2024-03-20T08:40:14.601 INFO:teuthology.orchestra.run.smithi183.stdout:/usr/bin/ceph: stdout "monmap": { 2024-03-20T08:40:14.601 INFO:teuthology.orchestra.run.smithi183.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-20T08:40:14.601 INFO:teuthology.orchestra.run.smithi183.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-03-20T08:40:14.601 INFO:teuthology.orchestra.run.smithi183.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-03-20T08:40:14.601 INFO:teuthology.orchestra.run.smithi183.stdout:/usr/bin/ceph: stdout }, 2024-03-20T08:40:14.601 INFO:teuthology.orchestra.run.smithi183.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-03-20T08:40:14.601 INFO:teuthology.orchestra.run.smithi183.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-20T08:40:14.601 INFO:teuthology.orchestra.run.smithi183.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-03-20T08:40:14.602 INFO:teuthology.orchestra.run.smithi183.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-03-20T08:40:14.602 INFO:teuthology.orchestra.run.smithi183.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-03-20T08:40:14.602 INFO:teuthology.orchestra.run.smithi183.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-03-20T08:40:14.602 INFO:teuthology.orchestra.run.smithi183.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-03-20T08:40:14.602 INFO:teuthology.orchestra.run.smithi183.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-03-20T08:40:14.605 INFO:teuthology.orchestra.run.smithi183.stdout:/usr/bin/ceph: stdout }, 2024-03-20T08:40:14.605 INFO:teuthology.orchestra.run.smithi183.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-03-20T08:40:14.605 INFO:teuthology.orchestra.run.smithi183.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-03-20T08:40:14.605 INFO:teuthology.orchestra.run.smithi183.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-03-20T08:40:14.605 INFO:teuthology.orchestra.run.smithi183.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-03-20T08:40:14.605 INFO:teuthology.orchestra.run.smithi183.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-03-20T08:40:14.605 INFO:teuthology.orchestra.run.smithi183.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-03-20T08:40:14.606 INFO:teuthology.orchestra.run.smithi183.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-03-20T08:40:14.606 INFO:teuthology.orchestra.run.smithi183.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-03-20T08:40:14.606 INFO:teuthology.orchestra.run.smithi183.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-03-20T08:40:14.606 INFO:teuthology.orchestra.run.smithi183.stdout:/usr/bin/ceph: stdout }, 2024-03-20T08:40:14.606 INFO:teuthology.orchestra.run.smithi183.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-03-20T08:40:14.606 INFO:teuthology.orchestra.run.smithi183.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-20T08:40:14.606 INFO:teuthology.orchestra.run.smithi183.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-03-20T08:40:14.606 INFO:teuthology.orchestra.run.smithi183.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-03-20T08:40:14.606 INFO:teuthology.orchestra.run.smithi183.stdout:/usr/bin/ceph: stdout }, 2024-03-20T08:40:14.606 INFO:teuthology.orchestra.run.smithi183.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-03-20T08:40:14.606 INFO:teuthology.orchestra.run.smithi183.stdout:/usr/bin/ceph: stdout "available": false, 2024-03-20T08:40:14.606 INFO:teuthology.orchestra.run.smithi183.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-03-20T08:40:14.606 INFO:teuthology.orchestra.run.smithi183.stdout:/usr/bin/ceph: stdout "modules": [ 2024-03-20T08:40:14.606 INFO:teuthology.orchestra.run.smithi183.stdout:/usr/bin/ceph: stdout "iostat", 2024-03-20T08:40:14.606 INFO:teuthology.orchestra.run.smithi183.stdout:/usr/bin/ceph: stdout "nfs", 2024-03-20T08:40:14.607 INFO:teuthology.orchestra.run.smithi183.stdout:/usr/bin/ceph: stdout "restful" 2024-03-20T08:40:14.607 INFO:teuthology.orchestra.run.smithi183.stdout:/usr/bin/ceph: stdout ], 2024-03-20T08:40:14.607 INFO:teuthology.orchestra.run.smithi183.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-20T08:40:14.607 INFO:teuthology.orchestra.run.smithi183.stdout:/usr/bin/ceph: stdout }, 2024-03-20T08:40:14.607 INFO:teuthology.orchestra.run.smithi183.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-03-20T08:40:14.607 INFO:teuthology.orchestra.run.smithi183.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-20T08:40:14.607 INFO:teuthology.orchestra.run.smithi183.stdout:/usr/bin/ceph: stdout "modified": "2024-03-20T08:40:04.332842+0000", 2024-03-20T08:40:14.607 INFO:teuthology.orchestra.run.smithi183.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-20T08:40:14.607 INFO:teuthology.orchestra.run.smithi183.stdout:/usr/bin/ceph: stdout }, 2024-03-20T08:40:14.607 INFO:teuthology.orchestra.run.smithi183.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-03-20T08:40:14.607 INFO:teuthology.orchestra.run.smithi183.stdout:/usr/bin/ceph: stdout } 2024-03-20T08:40:14.607 INFO:teuthology.orchestra.run.smithi183.stdout:mgr not available, waiting (1/15)... 2024-03-20T08:40:15.115 INFO:journalctl@ceph.mgr.y.smithi183.stdout:Mar 20 08:40:14 smithi183 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-mgr-y[25513]: 2024-03-20T08:40:14.788+0000 7f2a95edb200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-20T08:40:15.115 INFO:journalctl@ceph.mgr.y.smithi183.stdout:Mar 20 08:40:14 smithi183 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-mgr-y[25513]: 2024-03-20T08:40:14.879+0000 7f2a95edb200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-20T08:40:15.115 INFO:journalctl@ceph.mgr.y.smithi183.stdout:Mar 20 08:40:15 smithi183 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-mgr-y[25513]: 2024-03-20T08:40:15.020+0000 7f2a95edb200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-20T08:40:15.454 INFO:journalctl@ceph.mgr.y.smithi183.stdout:Mar 20 08:40:15 smithi183 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-mgr-y[25513]: 2024-03-20T08:40:15.115+0000 7f2a95edb200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-20T08:40:15.454 INFO:journalctl@ceph.mgr.y.smithi183.stdout:Mar 20 08:40:15 smithi183 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-mgr-y[25513]: 2024-03-20T08:40:15.197+0000 7f2a95edb200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-20T08:40:15.745 INFO:journalctl@ceph.mgr.y.smithi183.stdout:Mar 20 08:40:15 smithi183 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-mgr-y[25513]: 2024-03-20T08:40:15.459+0000 7f2a95edb200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-20T08:40:15.746 INFO:journalctl@ceph.mgr.y.smithi183.stdout:Mar 20 08:40:15 smithi183 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-mgr-y[25513]: 2024-03-20T08:40:15.567+0000 7f2a95edb200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-20T08:40:15.746 INFO:journalctl@ceph.mgr.y.smithi183.stdout:Mar 20 08:40:15 smithi183 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-mgr-y[25513]: 2024-03-20T08:40:15.652+0000 7f2a95edb200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-20T08:40:15.746 INFO:journalctl@ceph.mgr.y.smithi183.stdout:Mar 20 08:40:15 smithi183 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-mgr-y[25513]: 2024-03-20T08:40:15.745+0000 7f2a95edb200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-20T08:40:16.108 INFO:journalctl@ceph.mgr.y.smithi183.stdout:Mar 20 08:40:15 smithi183 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-mgr-y[25513]: 2024-03-20T08:40:15.921+0000 7f2a95edb200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-20T08:40:16.954 INFO:journalctl@ceph.mgr.y.smithi183.stdout:Mar 20 08:40:16 smithi183 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-mgr-y[25513]: 2024-03-20T08:40:16.912+0000 7f2a95edb200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-20T08:40:17.417 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:40:17 smithi183 ceph-mon[25285]: from='client.? 172.21.15.183:0/4035672189' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-03-20T08:40:17.418 INFO:journalctl@ceph.mgr.y.smithi183.stdout:Mar 20 08:40:17 smithi183 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-mgr-y[25513]: 2024-03-20T08:40:17.048+0000 7f2a95edb200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-20T08:40:17.418 INFO:journalctl@ceph.mgr.y.smithi183.stdout:Mar 20 08:40:17 smithi183 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-mgr-y[25513]: 2024-03-20T08:40:17.390+0000 7f2a95edb200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-20T08:40:17.675 INFO:journalctl@ceph.mgr.y.smithi183.stdout:Mar 20 08:40:17 smithi183 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-mgr-y[25513]: 2024-03-20T08:40:17.480+0000 7f2a95edb200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-20T08:40:17.954 INFO:journalctl@ceph.mgr.y.smithi183.stdout:Mar 20 08:40:17 smithi183 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-mgr-y[25513]: 2024-03-20T08:40:17.675+0000 7f2a95edb200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-20T08:40:17.954 INFO:journalctl@ceph.mgr.y.smithi183.stdout:Mar 20 08:40:17 smithi183 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-mgr-y[25513]: 2024-03-20T08:40:17.859+0000 7f2a95edb200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-20T08:40:18.065 INFO:teuthology.orchestra.run.smithi183.stdout:/usr/bin/ceph: stdout 2024-03-20T08:40:18.065 INFO:teuthology.orchestra.run.smithi183.stdout:/usr/bin/ceph: stdout { 2024-03-20T08:40:18.065 INFO:teuthology.orchestra.run.smithi183.stdout:/usr/bin/ceph: stdout "fsid": "4c3f3eba-e695-11ee-95c9-87774f69a715", 2024-03-20T08:40:18.065 INFO:teuthology.orchestra.run.smithi183.stdout:/usr/bin/ceph: stdout "health": { 2024-03-20T08:40:18.065 INFO:teuthology.orchestra.run.smithi183.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-03-20T08:40:18.065 INFO:teuthology.orchestra.run.smithi183.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-03-20T08:40:18.065 INFO:teuthology.orchestra.run.smithi183.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-03-20T08:40:18.065 INFO:teuthology.orchestra.run.smithi183.stdout:/usr/bin/ceph: stdout }, 2024-03-20T08:40:18.065 INFO:teuthology.orchestra.run.smithi183.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-03-20T08:40:18.065 INFO:teuthology.orchestra.run.smithi183.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-03-20T08:40:18.065 INFO:teuthology.orchestra.run.smithi183.stdout:/usr/bin/ceph: stdout 0 2024-03-20T08:40:18.066 INFO:teuthology.orchestra.run.smithi183.stdout:/usr/bin/ceph: stdout ], 2024-03-20T08:40:18.066 INFO:teuthology.orchestra.run.smithi183.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-03-20T08:40:18.066 INFO:teuthology.orchestra.run.smithi183.stdout:/usr/bin/ceph: stdout "a" 2024-03-20T08:40:18.066 INFO:teuthology.orchestra.run.smithi183.stdout:/usr/bin/ceph: stdout ], 2024-03-20T08:40:18.066 INFO:teuthology.orchestra.run.smithi183.stdout:/usr/bin/ceph: stdout "quorum_age": 6, 2024-03-20T08:40:18.066 INFO:teuthology.orchestra.run.smithi183.stdout:/usr/bin/ceph: stdout "monmap": { 2024-03-20T08:40:18.066 INFO:teuthology.orchestra.run.smithi183.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-20T08:40:18.066 INFO:teuthology.orchestra.run.smithi183.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-03-20T08:40:18.066 INFO:teuthology.orchestra.run.smithi183.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-03-20T08:40:18.066 INFO:teuthology.orchestra.run.smithi183.stdout:/usr/bin/ceph: stdout }, 2024-03-20T08:40:18.066 INFO:teuthology.orchestra.run.smithi183.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-03-20T08:40:18.066 INFO:teuthology.orchestra.run.smithi183.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-20T08:40:18.066 INFO:teuthology.orchestra.run.smithi183.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-03-20T08:40:18.066 INFO:teuthology.orchestra.run.smithi183.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-03-20T08:40:18.066 INFO:teuthology.orchestra.run.smithi183.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-03-20T08:40:18.067 INFO:teuthology.orchestra.run.smithi183.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-03-20T08:40:18.067 INFO:teuthology.orchestra.run.smithi183.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-03-20T08:40:18.067 INFO:teuthology.orchestra.run.smithi183.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-03-20T08:40:18.070 INFO:teuthology.orchestra.run.smithi183.stdout:/usr/bin/ceph: stdout }, 2024-03-20T08:40:18.070 INFO:teuthology.orchestra.run.smithi183.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-03-20T08:40:18.071 INFO:teuthology.orchestra.run.smithi183.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-03-20T08:40:18.071 INFO:teuthology.orchestra.run.smithi183.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-03-20T08:40:18.071 INFO:teuthology.orchestra.run.smithi183.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-03-20T08:40:18.071 INFO:teuthology.orchestra.run.smithi183.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-03-20T08:40:18.071 INFO:teuthology.orchestra.run.smithi183.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-03-20T08:40:18.071 INFO:teuthology.orchestra.run.smithi183.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-03-20T08:40:18.071 INFO:teuthology.orchestra.run.smithi183.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-03-20T08:40:18.071 INFO:teuthology.orchestra.run.smithi183.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-03-20T08:40:18.071 INFO:teuthology.orchestra.run.smithi183.stdout:/usr/bin/ceph: stdout }, 2024-03-20T08:40:18.071 INFO:teuthology.orchestra.run.smithi183.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-03-20T08:40:18.071 INFO:teuthology.orchestra.run.smithi183.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-20T08:40:18.072 INFO:teuthology.orchestra.run.smithi183.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-03-20T08:40:18.072 INFO:teuthology.orchestra.run.smithi183.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-03-20T08:40:18.072 INFO:teuthology.orchestra.run.smithi183.stdout:/usr/bin/ceph: stdout }, 2024-03-20T08:40:18.072 INFO:teuthology.orchestra.run.smithi183.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-03-20T08:40:18.072 INFO:teuthology.orchestra.run.smithi183.stdout:/usr/bin/ceph: stdout "available": false, 2024-03-20T08:40:18.072 INFO:teuthology.orchestra.run.smithi183.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-03-20T08:40:18.072 INFO:teuthology.orchestra.run.smithi183.stdout:/usr/bin/ceph: stdout "modules": [ 2024-03-20T08:40:18.072 INFO:teuthology.orchestra.run.smithi183.stdout:/usr/bin/ceph: stdout "iostat", 2024-03-20T08:40:18.072 INFO:teuthology.orchestra.run.smithi183.stdout:/usr/bin/ceph: stdout "nfs", 2024-03-20T08:40:18.072 INFO:teuthology.orchestra.run.smithi183.stdout:/usr/bin/ceph: stdout "restful" 2024-03-20T08:40:18.072 INFO:teuthology.orchestra.run.smithi183.stdout:/usr/bin/ceph: stdout ], 2024-03-20T08:40:18.072 INFO:teuthology.orchestra.run.smithi183.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-20T08:40:18.072 INFO:teuthology.orchestra.run.smithi183.stdout:/usr/bin/ceph: stdout }, 2024-03-20T08:40:18.072 INFO:teuthology.orchestra.run.smithi183.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-03-20T08:40:18.072 INFO:teuthology.orchestra.run.smithi183.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-20T08:40:18.072 INFO:teuthology.orchestra.run.smithi183.stdout:/usr/bin/ceph: stdout "modified": "2024-03-20T08:40:04.332842+0000", 2024-03-20T08:40:18.073 INFO:teuthology.orchestra.run.smithi183.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-20T08:40:18.073 INFO:teuthology.orchestra.run.smithi183.stdout:/usr/bin/ceph: stdout }, 2024-03-20T08:40:18.073 INFO:teuthology.orchestra.run.smithi183.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-03-20T08:40:18.073 INFO:teuthology.orchestra.run.smithi183.stdout:/usr/bin/ceph: stdout } 2024-03-20T08:40:18.073 INFO:teuthology.orchestra.run.smithi183.stdout:mgr not available, waiting (2/15)... 2024-03-20T08:40:18.406 INFO:journalctl@ceph.mgr.y.smithi183.stdout:Mar 20 08:40:18 smithi183 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-mgr-y[25513]: 2024-03-20T08:40:18.143+0000 7f2a95edb200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-20T08:40:18.407 INFO:journalctl@ceph.mgr.y.smithi183.stdout:Mar 20 08:40:18 smithi183 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-mgr-y[25513]: 2024-03-20T08:40:18.406+0000 7f2a95edb200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-20T08:40:18.704 INFO:journalctl@ceph.mgr.y.smithi183.stdout:Mar 20 08:40:18 smithi183 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-mgr-y[25513]: 2024-03-20T08:40:18.641+0000 7f2a95edb200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-20T08:40:19.704 INFO:journalctl@ceph.mgr.y.smithi183.stdout:Mar 20 08:40:19 smithi183 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-mgr-y[25513]: 2024-03-20T08:40:19.439+0000 7f2a95edb200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-20T08:40:20.290 INFO:journalctl@ceph.mgr.y.smithi183.stdout:Mar 20 08:40:20 smithi183 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-mgr-y[25513]: 2024-03-20T08:40:20.008+0000 7f2a95edb200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-20T08:40:20.291 INFO:journalctl@ceph.mgr.y.smithi183.stdout:Mar 20 08:40:20 smithi183 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-mgr-y[25513]: 2024-03-20T08:40:20.109+0000 7f2a95edb200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-20T08:40:20.558 INFO:journalctl@ceph.mgr.y.smithi183.stdout:Mar 20 08:40:20 smithi183 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-mgr-y[25513]: 2024-03-20T08:40:20.328+0000 7f2a95edb200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-20T08:40:20.559 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:40:20 smithi183 ceph-mon[25285]: Activating manager daemon y 2024-03-20T08:40:20.559 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:40:20 smithi183 ceph-mon[25285]: mgrmap e2: y(active, starting, since 0.00296917s) 2024-03-20T08:40:20.559 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:40:20 smithi183 ceph-mon[25285]: from='mgr.14100 172.21.15.183:0/893187882' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-20T08:40:20.559 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:40:20 smithi183 ceph-mon[25285]: from='mgr.14100 172.21.15.183:0/893187882' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-20T08:40:20.559 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:40:20 smithi183 ceph-mon[25285]: from='mgr.14100 172.21.15.183:0/893187882' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-20T08:40:20.559 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:40:20 smithi183 ceph-mon[25285]: from='mgr.14100 172.21.15.183:0/893187882' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-20T08:40:20.559 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:40:20 smithi183 ceph-mon[25285]: from='mgr.14100 172.21.15.183:0/893187882' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-20T08:40:20.559 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:40:20 smithi183 ceph-mon[25285]: Manager daemon y is now available 2024-03-20T08:40:20.559 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:40:20 smithi183 ceph-mon[25285]: from='mgr.14100 172.21.15.183:0/893187882' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-20T08:40:20.559 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:40:20 smithi183 ceph-mon[25285]: from='mgr.14100 172.21.15.183:0/893187882' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-20T08:40:20.560 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:40:20 smithi183 ceph-mon[25285]: from='mgr.14100 172.21.15.183:0/893187882' entity='mgr.y' 2024-03-20T08:40:20.560 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:40:20 smithi183 ceph-mon[25285]: from='mgr.14100 172.21.15.183:0/893187882' entity='mgr.y' 2024-03-20T08:40:20.560 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:40:20 smithi183 ceph-mon[25285]: from='mgr.14100 172.21.15.183:0/893187882' entity='mgr.y' 2024-03-20T08:40:21.039 INFO:teuthology.orchestra.run.smithi183.stdout:/usr/bin/ceph: stdout 2024-03-20T08:40:21.039 INFO:teuthology.orchestra.run.smithi183.stdout:/usr/bin/ceph: stdout { 2024-03-20T08:40:21.039 INFO:teuthology.orchestra.run.smithi183.stdout:/usr/bin/ceph: stdout "fsid": "4c3f3eba-e695-11ee-95c9-87774f69a715", 2024-03-20T08:40:21.039 INFO:teuthology.orchestra.run.smithi183.stdout:/usr/bin/ceph: stdout "health": { 2024-03-20T08:40:21.039 INFO:teuthology.orchestra.run.smithi183.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-03-20T08:40:21.040 INFO:teuthology.orchestra.run.smithi183.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-03-20T08:40:21.040 INFO:teuthology.orchestra.run.smithi183.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-03-20T08:40:21.040 INFO:teuthology.orchestra.run.smithi183.stdout:/usr/bin/ceph: stdout }, 2024-03-20T08:40:21.040 INFO:teuthology.orchestra.run.smithi183.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-03-20T08:40:21.040 INFO:teuthology.orchestra.run.smithi183.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-03-20T08:40:21.040 INFO:teuthology.orchestra.run.smithi183.stdout:/usr/bin/ceph: stdout 0 2024-03-20T08:40:21.040 INFO:teuthology.orchestra.run.smithi183.stdout:/usr/bin/ceph: stdout ], 2024-03-20T08:40:21.040 INFO:teuthology.orchestra.run.smithi183.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-03-20T08:40:21.040 INFO:teuthology.orchestra.run.smithi183.stdout:/usr/bin/ceph: stdout "a" 2024-03-20T08:40:21.040 INFO:teuthology.orchestra.run.smithi183.stdout:/usr/bin/ceph: stdout ], 2024-03-20T08:40:21.040 INFO:teuthology.orchestra.run.smithi183.stdout:/usr/bin/ceph: stdout "quorum_age": 10, 2024-03-20T08:40:21.040 INFO:teuthology.orchestra.run.smithi183.stdout:/usr/bin/ceph: stdout "monmap": { 2024-03-20T08:40:21.040 INFO:teuthology.orchestra.run.smithi183.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-20T08:40:21.040 INFO:teuthology.orchestra.run.smithi183.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-03-20T08:40:21.040 INFO:teuthology.orchestra.run.smithi183.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-03-20T08:40:21.041 INFO:teuthology.orchestra.run.smithi183.stdout:/usr/bin/ceph: stdout }, 2024-03-20T08:40:21.041 INFO:teuthology.orchestra.run.smithi183.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-03-20T08:40:21.041 INFO:teuthology.orchestra.run.smithi183.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-20T08:40:21.041 INFO:teuthology.orchestra.run.smithi183.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-03-20T08:40:21.041 INFO:teuthology.orchestra.run.smithi183.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-03-20T08:40:21.041 INFO:teuthology.orchestra.run.smithi183.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-03-20T08:40:21.041 INFO:teuthology.orchestra.run.smithi183.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-03-20T08:40:21.041 INFO:teuthology.orchestra.run.smithi183.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-03-20T08:40:21.041 INFO:teuthology.orchestra.run.smithi183.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-03-20T08:40:21.041 INFO:teuthology.orchestra.run.smithi183.stdout:/usr/bin/ceph: stdout }, 2024-03-20T08:40:21.041 INFO:teuthology.orchestra.run.smithi183.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-03-20T08:40:21.041 INFO:teuthology.orchestra.run.smithi183.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-03-20T08:40:21.041 INFO:teuthology.orchestra.run.smithi183.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-03-20T08:40:21.041 INFO:teuthology.orchestra.run.smithi183.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-03-20T08:40:21.042 INFO:teuthology.orchestra.run.smithi183.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-03-20T08:40:21.042 INFO:teuthology.orchestra.run.smithi183.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-03-20T08:40:21.042 INFO:teuthology.orchestra.run.smithi183.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-03-20T08:40:21.042 INFO:teuthology.orchestra.run.smithi183.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-03-20T08:40:21.042 INFO:teuthology.orchestra.run.smithi183.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-03-20T08:40:21.042 INFO:teuthology.orchestra.run.smithi183.stdout:/usr/bin/ceph: stdout }, 2024-03-20T08:40:21.042 INFO:teuthology.orchestra.run.smithi183.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-03-20T08:40:21.042 INFO:teuthology.orchestra.run.smithi183.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-20T08:40:21.042 INFO:teuthology.orchestra.run.smithi183.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-03-20T08:40:21.043 INFO:teuthology.orchestra.run.smithi183.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-03-20T08:40:21.043 INFO:teuthology.orchestra.run.smithi183.stdout:/usr/bin/ceph: stdout }, 2024-03-20T08:40:21.043 INFO:teuthology.orchestra.run.smithi183.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-03-20T08:40:21.043 INFO:teuthology.orchestra.run.smithi183.stdout:/usr/bin/ceph: stdout "available": false, 2024-03-20T08:40:21.043 INFO:teuthology.orchestra.run.smithi183.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-03-20T08:40:21.043 INFO:teuthology.orchestra.run.smithi183.stdout:/usr/bin/ceph: stdout "modules": [ 2024-03-20T08:40:21.043 INFO:teuthology.orchestra.run.smithi183.stdout:/usr/bin/ceph: stdout "iostat", 2024-03-20T08:40:21.043 INFO:teuthology.orchestra.run.smithi183.stdout:/usr/bin/ceph: stdout "nfs", 2024-03-20T08:40:21.043 INFO:teuthology.orchestra.run.smithi183.stdout:/usr/bin/ceph: stdout "restful" 2024-03-20T08:40:21.043 INFO:teuthology.orchestra.run.smithi183.stdout:/usr/bin/ceph: stdout ], 2024-03-20T08:40:21.043 INFO:teuthology.orchestra.run.smithi183.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-20T08:40:21.043 INFO:teuthology.orchestra.run.smithi183.stdout:/usr/bin/ceph: stdout }, 2024-03-20T08:40:21.044 INFO:teuthology.orchestra.run.smithi183.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-03-20T08:40:21.044 INFO:teuthology.orchestra.run.smithi183.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-20T08:40:21.044 INFO:teuthology.orchestra.run.smithi183.stdout:/usr/bin/ceph: stdout "modified": "2024-03-20T08:40:04.332842+0000", 2024-03-20T08:40:21.044 INFO:teuthology.orchestra.run.smithi183.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-20T08:40:21.044 INFO:teuthology.orchestra.run.smithi183.stdout:/usr/bin/ceph: stdout }, 2024-03-20T08:40:21.044 INFO:teuthology.orchestra.run.smithi183.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-03-20T08:40:21.044 INFO:teuthology.orchestra.run.smithi183.stdout:/usr/bin/ceph: stdout } 2024-03-20T08:40:21.044 INFO:teuthology.orchestra.run.smithi183.stdout:mgr not available, waiting (3/15)... 2024-03-20T08:40:21.704 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:40:21 smithi183 ceph-mon[25285]: from='client.? 172.21.15.183:0/2744427127' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-03-20T08:40:21.704 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:40:21 smithi183 ceph-mon[25285]: mgrmap e3: y(active, since 1.00695s) 2024-03-20T08:40:23.426 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:40:23 smithi183 ceph-mon[25285]: mgrmap e4: y(active, since 2s) 2024-03-20T08:40:24.320 INFO:teuthology.orchestra.run.smithi183.stdout:/usr/bin/ceph: stdout 2024-03-20T08:40:24.320 INFO:teuthology.orchestra.run.smithi183.stdout:/usr/bin/ceph: stdout { 2024-03-20T08:40:24.320 INFO:teuthology.orchestra.run.smithi183.stdout:/usr/bin/ceph: stdout "fsid": "4c3f3eba-e695-11ee-95c9-87774f69a715", 2024-03-20T08:40:24.320 INFO:teuthology.orchestra.run.smithi183.stdout:/usr/bin/ceph: stdout "health": { 2024-03-20T08:40:24.320 INFO:teuthology.orchestra.run.smithi183.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-03-20T08:40:24.320 INFO:teuthology.orchestra.run.smithi183.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-03-20T08:40:24.320 INFO:teuthology.orchestra.run.smithi183.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-03-20T08:40:24.320 INFO:teuthology.orchestra.run.smithi183.stdout:/usr/bin/ceph: stdout }, 2024-03-20T08:40:24.321 INFO:teuthology.orchestra.run.smithi183.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-03-20T08:40:24.321 INFO:teuthology.orchestra.run.smithi183.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-03-20T08:40:24.321 INFO:teuthology.orchestra.run.smithi183.stdout:/usr/bin/ceph: stdout 0 2024-03-20T08:40:24.321 INFO:teuthology.orchestra.run.smithi183.stdout:/usr/bin/ceph: stdout ], 2024-03-20T08:40:24.321 INFO:teuthology.orchestra.run.smithi183.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-03-20T08:40:24.321 INFO:teuthology.orchestra.run.smithi183.stdout:/usr/bin/ceph: stdout "a" 2024-03-20T08:40:24.321 INFO:teuthology.orchestra.run.smithi183.stdout:/usr/bin/ceph: stdout ], 2024-03-20T08:40:24.321 INFO:teuthology.orchestra.run.smithi183.stdout:/usr/bin/ceph: stdout "quorum_age": 13, 2024-03-20T08:40:24.321 INFO:teuthology.orchestra.run.smithi183.stdout:/usr/bin/ceph: stdout "monmap": { 2024-03-20T08:40:24.321 INFO:teuthology.orchestra.run.smithi183.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-20T08:40:24.321 INFO:teuthology.orchestra.run.smithi183.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-03-20T08:40:24.321 INFO:teuthology.orchestra.run.smithi183.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-03-20T08:40:24.321 INFO:teuthology.orchestra.run.smithi183.stdout:/usr/bin/ceph: stdout }, 2024-03-20T08:40:24.321 INFO:teuthology.orchestra.run.smithi183.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-03-20T08:40:24.322 INFO:teuthology.orchestra.run.smithi183.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-20T08:40:24.322 INFO:teuthology.orchestra.run.smithi183.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-03-20T08:40:24.322 INFO:teuthology.orchestra.run.smithi183.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-03-20T08:40:24.322 INFO:teuthology.orchestra.run.smithi183.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-03-20T08:40:24.322 INFO:teuthology.orchestra.run.smithi183.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-03-20T08:40:24.322 INFO:teuthology.orchestra.run.smithi183.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-03-20T08:40:24.322 INFO:teuthology.orchestra.run.smithi183.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-03-20T08:40:24.322 INFO:teuthology.orchestra.run.smithi183.stdout:/usr/bin/ceph: stdout }, 2024-03-20T08:40:24.322 INFO:teuthology.orchestra.run.smithi183.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-03-20T08:40:24.322 INFO:teuthology.orchestra.run.smithi183.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-03-20T08:40:24.322 INFO:teuthology.orchestra.run.smithi183.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-03-20T08:40:24.322 INFO:teuthology.orchestra.run.smithi183.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-03-20T08:40:24.322 INFO:teuthology.orchestra.run.smithi183.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-03-20T08:40:24.322 INFO:teuthology.orchestra.run.smithi183.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-03-20T08:40:24.322 INFO:teuthology.orchestra.run.smithi183.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-03-20T08:40:24.323 INFO:teuthology.orchestra.run.smithi183.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-03-20T08:40:24.323 INFO:teuthology.orchestra.run.smithi183.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-03-20T08:40:24.323 INFO:teuthology.orchestra.run.smithi183.stdout:/usr/bin/ceph: stdout }, 2024-03-20T08:40:24.323 INFO:teuthology.orchestra.run.smithi183.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-03-20T08:40:24.323 INFO:teuthology.orchestra.run.smithi183.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-20T08:40:24.323 INFO:teuthology.orchestra.run.smithi183.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-03-20T08:40:24.323 INFO:teuthology.orchestra.run.smithi183.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-03-20T08:40:24.323 INFO:teuthology.orchestra.run.smithi183.stdout:/usr/bin/ceph: stdout }, 2024-03-20T08:40:24.323 INFO:teuthology.orchestra.run.smithi183.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-03-20T08:40:24.323 INFO:teuthology.orchestra.run.smithi183.stdout:/usr/bin/ceph: stdout "available": true, 2024-03-20T08:40:24.323 INFO:teuthology.orchestra.run.smithi183.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-03-20T08:40:24.323 INFO:teuthology.orchestra.run.smithi183.stdout:/usr/bin/ceph: stdout "modules": [ 2024-03-20T08:40:24.323 INFO:teuthology.orchestra.run.smithi183.stdout:/usr/bin/ceph: stdout "iostat", 2024-03-20T08:40:24.323 INFO:teuthology.orchestra.run.smithi183.stdout:/usr/bin/ceph: stdout "nfs", 2024-03-20T08:40:24.324 INFO:teuthology.orchestra.run.smithi183.stdout:/usr/bin/ceph: stdout "restful" 2024-03-20T08:40:24.324 INFO:teuthology.orchestra.run.smithi183.stdout:/usr/bin/ceph: stdout ], 2024-03-20T08:40:24.324 INFO:teuthology.orchestra.run.smithi183.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-20T08:40:24.324 INFO:teuthology.orchestra.run.smithi183.stdout:/usr/bin/ceph: stdout }, 2024-03-20T08:40:24.324 INFO:teuthology.orchestra.run.smithi183.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-03-20T08:40:24.324 INFO:teuthology.orchestra.run.smithi183.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-20T08:40:24.324 INFO:teuthology.orchestra.run.smithi183.stdout:/usr/bin/ceph: stdout "modified": "2024-03-20T08:40:04.332842+0000", 2024-03-20T08:40:24.324 INFO:teuthology.orchestra.run.smithi183.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-20T08:40:24.324 INFO:teuthology.orchestra.run.smithi183.stdout:/usr/bin/ceph: stdout }, 2024-03-20T08:40:24.324 INFO:teuthology.orchestra.run.smithi183.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-03-20T08:40:24.324 INFO:teuthology.orchestra.run.smithi183.stdout:/usr/bin/ceph: stdout } 2024-03-20T08:40:24.324 INFO:teuthology.orchestra.run.smithi183.stdout:mgr is available 2024-03-20T08:40:24.429 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:40:24 smithi183 ceph-mon[25285]: from='client.? 172.21.15.183:0/342634413' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-03-20T08:40:25.485 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:40:25 smithi183 ceph-mon[25285]: from='client.? 172.21.15.183:0/760650089' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-03-20T08:40:25.486 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:40:25 smithi183 ceph-mon[25285]: from='client.? 172.21.15.183:0/760650089' entity='client.admin' cmd='[{"prefix": "config assimilate-conf"}]': finished 2024-03-20T08:40:25.489 INFO:teuthology.orchestra.run.smithi183.stdout:/usr/bin/ceph: stdout 2024-03-20T08:40:25.489 INFO:teuthology.orchestra.run.smithi183.stdout:/usr/bin/ceph: stdout [global] 2024-03-20T08:40:25.489 INFO:teuthology.orchestra.run.smithi183.stdout:/usr/bin/ceph: stdout fsid = 4c3f3eba-e695-11ee-95c9-87774f69a715 2024-03-20T08:40:25.489 INFO:teuthology.orchestra.run.smithi183.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-03-20T08:40:25.489 INFO:teuthology.orchestra.run.smithi183.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-03-20T08:40:25.489 INFO:teuthology.orchestra.run.smithi183.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-03-20T08:40:25.489 INFO:teuthology.orchestra.run.smithi183.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-03-20T08:40:25.489 INFO:teuthology.orchestra.run.smithi183.stdout:/usr/bin/ceph: stdout 2024-03-20T08:40:25.490 INFO:teuthology.orchestra.run.smithi183.stdout:/usr/bin/ceph: stdout [mgr] 2024-03-20T08:40:25.490 INFO:teuthology.orchestra.run.smithi183.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-03-20T08:40:25.490 INFO:teuthology.orchestra.run.smithi183.stdout:/usr/bin/ceph: stdout 2024-03-20T08:40:25.490 INFO:teuthology.orchestra.run.smithi183.stdout:/usr/bin/ceph: stdout [osd] 2024-03-20T08:40:25.490 INFO:teuthology.orchestra.run.smithi183.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-03-20T08:40:25.490 INFO:teuthology.orchestra.run.smithi183.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-03-20T08:40:25.490 INFO:teuthology.orchestra.run.smithi183.stdout:Enabling cephadm module... 2024-03-20T08:40:26.657 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:40:26 smithi183 ceph-mon[25285]: from='client.? 172.21.15.183:0/3759645902' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "cephadm"}]: dispatch 2024-03-20T08:40:26.658 INFO:journalctl@ceph.mgr.y.smithi183.stdout:Mar 20 08:40:26 smithi183 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-mgr-y[25513]: ignoring --setuser ceph since I am not root 2024-03-20T08:40:26.658 INFO:journalctl@ceph.mgr.y.smithi183.stdout:Mar 20 08:40:26 smithi183 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-mgr-y[25513]: ignoring --setgroup ceph since I am not root 2024-03-20T08:40:27.204 INFO:journalctl@ceph.mgr.y.smithi183.stdout:Mar 20 08:40:26 smithi183 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-mgr-y[25513]: 2024-03-20T08:40:26.955+0000 7f4234417200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-20T08:40:27.204 INFO:journalctl@ceph.mgr.y.smithi183.stdout:Mar 20 08:40:27 smithi183 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-mgr-y[25513]: 2024-03-20T08:40:27.052+0000 7f4234417200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-20T08:40:27.628 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:40:27 smithi183 ceph-mon[25285]: from='client.? 172.21.15.183:0/3759645902' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "cephadm"}]': finished 2024-03-20T08:40:27.628 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:40:27 smithi183 ceph-mon[25285]: mgrmap e5: y(active, since 6s) 2024-03-20T08:40:27.914 INFO:journalctl@ceph.mgr.y.smithi183.stdout:Mar 20 08:40:27 smithi183 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-mgr-y[25513]: 2024-03-20T08:40:27.710+0000 7f4234417200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-20T08:40:27.914 INFO:journalctl@ceph.mgr.y.smithi183.stdout:Mar 20 08:40:27 smithi183 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-mgr-y[25513]: 2024-03-20T08:40:27.803+0000 7f4234417200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-20T08:40:28.142 INFO:teuthology.orchestra.run.smithi183.stdout:/usr/bin/ceph: stdout { 2024-03-20T08:40:28.142 INFO:teuthology.orchestra.run.smithi183.stdout:/usr/bin/ceph: stdout "epoch": 5, 2024-03-20T08:40:28.143 INFO:teuthology.orchestra.run.smithi183.stdout:/usr/bin/ceph: stdout "available": true, 2024-03-20T08:40:28.143 INFO:teuthology.orchestra.run.smithi183.stdout:/usr/bin/ceph: stdout "active_name": "y", 2024-03-20T08:40:28.143 INFO:teuthology.orchestra.run.smithi183.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-03-20T08:40:28.143 INFO:teuthology.orchestra.run.smithi183.stdout:/usr/bin/ceph: stdout } 2024-03-20T08:40:28.143 INFO:teuthology.orchestra.run.smithi183.stdout:Waiting for the mgr to restart... 2024-03-20T08:40:28.143 INFO:teuthology.orchestra.run.smithi183.stdout:Waiting for mgr epoch 5... 2024-03-20T08:40:28.204 INFO:journalctl@ceph.mgr.y.smithi183.stdout:Mar 20 08:40:27 smithi183 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-mgr-y[25513]: 2024-03-20T08:40:27.948+0000 7f4234417200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-20T08:40:28.204 INFO:journalctl@ceph.mgr.y.smithi183.stdout:Mar 20 08:40:28 smithi183 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-mgr-y[25513]: 2024-03-20T08:40:28.043+0000 7f4234417200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-20T08:40:28.204 INFO:journalctl@ceph.mgr.y.smithi183.stdout:Mar 20 08:40:28 smithi183 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-mgr-y[25513]: 2024-03-20T08:40:28.126+0000 7f4234417200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-20T08:40:28.512 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:40:28 smithi183 ceph-mon[25285]: from='client.? 172.21.15.183:0/4145198431' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-03-20T08:40:28.512 INFO:journalctl@ceph.mgr.y.smithi183.stdout:Mar 20 08:40:28 smithi183 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-mgr-y[25513]: 2024-03-20T08:40:28.400+0000 7f4234417200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-20T08:40:28.513 INFO:journalctl@ceph.mgr.y.smithi183.stdout:Mar 20 08:40:28 smithi183 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-mgr-y[25513]: 2024-03-20T08:40:28.512+0000 7f4234417200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-20T08:40:28.870 INFO:journalctl@ceph.mgr.y.smithi183.stdout:Mar 20 08:40:28 smithi183 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-mgr-y[25513]: 2024-03-20T08:40:28.601+0000 7f4234417200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-20T08:40:28.870 INFO:journalctl@ceph.mgr.y.smithi183.stdout:Mar 20 08:40:28 smithi183 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-mgr-y[25513]: 2024-03-20T08:40:28.694+0000 7f4234417200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-20T08:40:29.204 INFO:journalctl@ceph.mgr.y.smithi183.stdout:Mar 20 08:40:28 smithi183 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-mgr-y[25513]: 2024-03-20T08:40:28.869+0000 7f4234417200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-20T08:40:30.204 INFO:journalctl@ceph.mgr.y.smithi183.stdout:Mar 20 08:40:29 smithi183 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-mgr-y[25513]: 2024-03-20T08:40:29.813+0000 7f4234417200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-20T08:40:30.204 INFO:journalctl@ceph.mgr.y.smithi183.stdout:Mar 20 08:40:29 smithi183 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-mgr-y[25513]: 2024-03-20T08:40:29.941+0000 7f4234417200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-20T08:40:30.555 INFO:journalctl@ceph.mgr.y.smithi183.stdout:Mar 20 08:40:30 smithi183 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-mgr-y[25513]: 2024-03-20T08:40:30.275+0000 7f4234417200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-20T08:40:30.555 INFO:journalctl@ceph.mgr.y.smithi183.stdout:Mar 20 08:40:30 smithi183 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-mgr-y[25513]: 2024-03-20T08:40:30.363+0000 7f4234417200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-20T08:40:30.555 INFO:journalctl@ceph.mgr.y.smithi183.stdout:Mar 20 08:40:30 smithi183 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-mgr-y[25513]: 2024-03-20T08:40:30.554+0000 7f4234417200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-20T08:40:30.954 INFO:journalctl@ceph.mgr.y.smithi183.stdout:Mar 20 08:40:30 smithi183 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-mgr-y[25513]: 2024-03-20T08:40:30.738+0000 7f4234417200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-20T08:40:31.277 INFO:journalctl@ceph.mgr.y.smithi183.stdout:Mar 20 08:40:31 smithi183 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-mgr-y[25513]: 2024-03-20T08:40:31.017+0000 7f4234417200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-20T08:40:31.277 INFO:journalctl@ceph.mgr.y.smithi183.stdout:Mar 20 08:40:31 smithi183 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-mgr-y[25513]: 2024-03-20T08:40:31.276+0000 7f4234417200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-20T08:40:31.704 INFO:journalctl@ceph.mgr.y.smithi183.stdout:Mar 20 08:40:31 smithi183 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-mgr-y[25513]: 2024-03-20T08:40:31.508+0000 7f4234417200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-20T08:40:32.704 INFO:journalctl@ceph.mgr.y.smithi183.stdout:Mar 20 08:40:32 smithi183 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-mgr-y[25513]: 2024-03-20T08:40:32.296+0000 7f4234417200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-20T08:40:33.169 INFO:journalctl@ceph.mgr.y.smithi183.stdout:Mar 20 08:40:32 smithi183 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-mgr-y[25513]: 2024-03-20T08:40:32.860+0000 7f4234417200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-20T08:40:33.169 INFO:journalctl@ceph.mgr.y.smithi183.stdout:Mar 20 08:40:32 smithi183 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-mgr-y[25513]: 2024-03-20T08:40:32.956+0000 7f4234417200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-20T08:40:33.169 INFO:journalctl@ceph.mgr.y.smithi183.stdout:Mar 20 08:40:33 smithi183 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-mgr-y[25513]: 2024-03-20T08:40:33.168+0000 7f4234417200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-20T08:40:33.454 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:40:33 smithi183 ceph-mon[25285]: Active manager daemon y restarted 2024-03-20T08:40:33.454 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:40:33 smithi183 ceph-mon[25285]: Activating manager daemon y 2024-03-20T08:40:33.454 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:40:33 smithi183 ceph-mon[25285]: osdmap e2: 0 total, 0 up, 0 in 2024-03-20T08:40:33.454 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:40:33 smithi183 ceph-mon[25285]: mgrmap e6: y(active, starting, since 0.00339337s) 2024-03-20T08:40:33.454 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:40:33 smithi183 ceph-mon[25285]: from='mgr.14120 172.21.15.183:0/4190782725' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-20T08:40:33.454 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:40:33 smithi183 ceph-mon[25285]: from='mgr.14120 172.21.15.183:0/4190782725' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-20T08:40:33.454 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:40:33 smithi183 ceph-mon[25285]: from='mgr.14120 172.21.15.183:0/4190782725' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-20T08:40:33.455 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:40:33 smithi183 ceph-mon[25285]: from='mgr.14120 172.21.15.183:0/4190782725' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-20T08:40:33.455 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:40:33 smithi183 ceph-mon[25285]: from='mgr.14120 172.21.15.183:0/4190782725' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-20T08:40:33.455 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:40:33 smithi183 ceph-mon[25285]: Manager daemon y is now available 2024-03-20T08:40:33.455 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:40:33 smithi183 ceph-mon[25285]: from='mgr.14120 172.21.15.183:0/4190782725' entity='mgr.y' 2024-03-20T08:40:33.455 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:40:33 smithi183 ceph-mon[25285]: from='mgr.14120 172.21.15.183:0/4190782725' entity='mgr.y' 2024-03-20T08:40:33.455 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:40:33 smithi183 ceph-mon[25285]: from='mgr.14120 172.21.15.183:0/4190782725' entity='mgr.y' 2024-03-20T08:40:33.455 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:40:33 smithi183 ceph-mon[25285]: from='mgr.14120 172.21.15.183:0/4190782725' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-20T08:40:34.454 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:40:34 smithi183 ceph-mon[25285]: Found migration_current of "None". Setting to last migration. 2024-03-20T08:40:34.454 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:40:34 smithi183 ceph-mon[25285]: from='mgr.14120 172.21.15.183:0/4190782725' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-20T08:40:34.454 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:40:34 smithi183 ceph-mon[25285]: from='mgr.14120 172.21.15.183:0/4190782725' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-20T08:40:34.455 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:40:34 smithi183 ceph-mon[25285]: from='mgr.14120 172.21.15.183:0/4190782725' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-20T08:40:34.455 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:40:34 smithi183 ceph-mon[25285]: from='mgr.14120 172.21.15.183:0/4190782725' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-20T08:40:34.455 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:40:34 smithi183 ceph-mon[25285]: from='mgr.14120 172.21.15.183:0/4190782725' entity='mgr.y' 2024-03-20T08:40:34.455 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:40:34 smithi183 ceph-mon[25285]: from='mgr.14120 172.21.15.183:0/4190782725' entity='mgr.y' 2024-03-20T08:40:34.455 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:40:34 smithi183 ceph-mon[25285]: mgrmap e7: y(active, since 1.0052s) 2024-03-20T08:40:34.747 INFO:teuthology.orchestra.run.smithi183.stdout:/usr/bin/ceph: stdout { 2024-03-20T08:40:34.748 INFO:teuthology.orchestra.run.smithi183.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 7, 2024-03-20T08:40:34.748 INFO:teuthology.orchestra.run.smithi183.stdout:/usr/bin/ceph: stdout "initialized": true 2024-03-20T08:40:34.748 INFO:teuthology.orchestra.run.smithi183.stdout:/usr/bin/ceph: stdout } 2024-03-20T08:40:34.748 INFO:teuthology.orchestra.run.smithi183.stdout:mgr epoch 5 is available 2024-03-20T08:40:34.748 INFO:teuthology.orchestra.run.smithi183.stdout:Setting orchestrator backend to cephadm... 2024-03-20T08:40:35.381 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:40:35 smithi183 ceph-mon[25285]: [20/Mar/2024:08:40:34] ENGINE Bus STARTING 2024-03-20T08:40:35.381 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:40:35 smithi183 ceph-mon[25285]: [20/Mar/2024:08:40:34] ENGINE Serving on https://172.21.15.183:7150 2024-03-20T08:40:35.382 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:40:35 smithi183 ceph-mon[25285]: from='client.14124 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-03-20T08:40:35.382 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:40:35 smithi183 ceph-mon[25285]: from='client.14124 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-03-20T08:40:35.382 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:40:35 smithi183 ceph-mon[25285]: [20/Mar/2024:08:40:34] ENGINE Serving on http://172.21.15.183:8765 2024-03-20T08:40:35.382 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:40:35 smithi183 ceph-mon[25285]: [20/Mar/2024:08:40:34] ENGINE Bus STARTED 2024-03-20T08:40:35.382 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:40:35 smithi183 ceph-mon[25285]: from='mgr.14120 172.21.15.183:0/4190782725' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-20T08:40:36.599 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:40:36 smithi183 ceph-mon[25285]: mgrmap e8: y(active, since 2s) 2024-03-20T08:40:36.600 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:40:36 smithi183 ceph-mon[25285]: from='client.14132 -' entity='client.admin' cmd=[{"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}]: dispatch 2024-03-20T08:40:36.600 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:40:36 smithi183 ceph-mon[25285]: from='mgr.14120 172.21.15.183:0/4190782725' entity='mgr.y' 2024-03-20T08:40:36.600 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:40:36 smithi183 ceph-mon[25285]: from='mgr.14120 172.21.15.183:0/4190782725' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-20T08:40:37.226 INFO:teuthology.orchestra.run.smithi183.stdout:/usr/bin/ceph: stdout value unchanged 2024-03-20T08:40:37.227 INFO:teuthology.orchestra.run.smithi183.stdout:Generating ssh key... 2024-03-20T08:40:37.954 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:40:37 smithi183 ceph-mon[25285]: from='client.14134 -' entity='client.admin' cmd=[{"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}]: dispatch 2024-03-20T08:40:38.313 INFO:journalctl@ceph.mgr.y.smithi183.stdout:Mar 20 08:40:38 smithi183 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-mgr-y[25513]: Generating public/private rsa key pair. 2024-03-20T08:40:38.313 INFO:journalctl@ceph.mgr.y.smithi183.stdout:Mar 20 08:40:38 smithi183 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-mgr-y[25513]: Your identification has been saved in /tmp/tmpf0ext_8t/key. 2024-03-20T08:40:38.313 INFO:journalctl@ceph.mgr.y.smithi183.stdout:Mar 20 08:40:38 smithi183 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-mgr-y[25513]: Your public key has been saved in /tmp/tmpf0ext_8t/key.pub. 2024-03-20T08:40:38.313 INFO:journalctl@ceph.mgr.y.smithi183.stdout:Mar 20 08:40:38 smithi183 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-mgr-y[25513]: The key fingerprint is: 2024-03-20T08:40:38.313 INFO:journalctl@ceph.mgr.y.smithi183.stdout:Mar 20 08:40:38 smithi183 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-mgr-y[25513]: SHA256:8QiTjkJMw5pC+lnkXJZzmAl3vpd/GiVdl++353Ilr60 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715 2024-03-20T08:40:38.313 INFO:journalctl@ceph.mgr.y.smithi183.stdout:Mar 20 08:40:38 smithi183 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-mgr-y[25513]: The key's randomart image is: 2024-03-20T08:40:38.313 INFO:journalctl@ceph.mgr.y.smithi183.stdout:Mar 20 08:40:38 smithi183 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-mgr-y[25513]: +---[RSA 3072]----+ 2024-03-20T08:40:38.313 INFO:journalctl@ceph.mgr.y.smithi183.stdout:Mar 20 08:40:38 smithi183 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-mgr-y[25513]: | .o ...=. | 2024-03-20T08:40:38.313 INFO:journalctl@ceph.mgr.y.smithi183.stdout:Mar 20 08:40:38 smithi183 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-mgr-y[25513]: | +....O+. .| 2024-03-20T08:40:38.314 INFO:journalctl@ceph.mgr.y.smithi183.stdout:Mar 20 08:40:38 smithi183 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-mgr-y[25513]: |ooo+ o+oo .o| 2024-03-20T08:40:38.314 INFO:journalctl@ceph.mgr.y.smithi183.stdout:Mar 20 08:40:38 smithi183 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-mgr-y[25513]: |=. +o o = . . .o| 2024-03-20T08:40:38.314 INFO:journalctl@ceph.mgr.y.smithi183.stdout:Mar 20 08:40:38 smithi183 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-mgr-y[25513]: |...o. . S + . o .| 2024-03-20T08:40:38.314 INFO:journalctl@ceph.mgr.y.smithi183.stdout:Mar 20 08:40:38 smithi183 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-mgr-y[25513]: | o. . . o...| 2024-03-20T08:40:38.314 INFO:journalctl@ceph.mgr.y.smithi183.stdout:Mar 20 08:40:38 smithi183 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-mgr-y[25513]: | o .o+| 2024-03-20T08:40:38.314 INFO:journalctl@ceph.mgr.y.smithi183.stdout:Mar 20 08:40:38 smithi183 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-mgr-y[25513]: | +..*| 2024-03-20T08:40:38.314 INFO:journalctl@ceph.mgr.y.smithi183.stdout:Mar 20 08:40:38 smithi183 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-mgr-y[25513]: | . EB+| 2024-03-20T08:40:38.314 INFO:journalctl@ceph.mgr.y.smithi183.stdout:Mar 20 08:40:38 smithi183 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-mgr-y[25513]: +----[SHA256]-----+ 2024-03-20T08:40:39.175 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:40:39 smithi183 ceph-mon[25285]: from='client.14136 -' entity='client.admin' cmd=[{"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}]: dispatch 2024-03-20T08:40:39.176 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:40:39 smithi183 ceph-mon[25285]: Generating ssh key... 2024-03-20T08:40:39.176 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:40:39 smithi183 ceph-mon[25285]: from='mgr.14120 172.21.15.183:0/4190782725' entity='mgr.y' 2024-03-20T08:40:39.176 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:40:39 smithi183 ceph-mon[25285]: from='mgr.14120 172.21.15.183:0/4190782725' entity='mgr.y' 2024-03-20T08:40:39.756 INFO:teuthology.orchestra.run.smithi183.stdout:/usr/bin/ceph: stdout ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCu7rXZCzuPfK508y3lsFrV1JQq4dlVbN73hCrA76eus1RG6/erFzIM9L71J2LAa+HceWm7jIhwtGTJlzGuJgG4fgDirQNTb+w2hU0XDLBmwr1pNO5f2BdvPpi7t26qUwhLs6P/zmEsyyoxL/iqelDcR/vb/FUq5r6UF/TIFGKwKkiuNRyREhquxGP7gXuMevi/PKeM+bDoYmoCfZ3TGGjgrfi65YifYEomcxsdzQJBB2BLChkOkwNQBMsEKYZo4RhBgpcGJA7IHk7cH+lRZxDFLzglEFEd5aqSTw97TQAgjwnD4bw03mIMTMw+RKoxmZWjCMLqTfYjze1SiJOIA4dqplIXHiUMB8evBa0b7j5APr0O47iwz1jdvuTecipAN0vxxTh0scdoUEk7lgOhE+eb2qPyrs0lxSiirkeFJDOMDpmiFJmb01/VHEgwrgJT1W21FE17O8Y8Z/E9M2U7A7r6XlLtQyIZlD7oPzVcnNdweatZWvo6cma8tvvdZPaMI98= ceph-4c3f3eba-e695-11ee-95c9-87774f69a715 2024-03-20T08:40:39.756 INFO:teuthology.orchestra.run.smithi183.stdout:Wrote public SSH key to /home/ubuntu/cephtest/ceph.pub 2024-03-20T08:40:39.756 INFO:teuthology.orchestra.run.smithi183.stdout:Adding key to root@localhost authorized_keys... 2024-03-20T08:40:39.757 INFO:teuthology.orchestra.run.smithi183.stdout:Adding host smithi183... 2024-03-20T08:40:40.349 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:40:40 smithi183 ceph-mon[25285]: from='client.14138 -' entity='client.admin' cmd=[{"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}]: dispatch 2024-03-20T08:40:41.270 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:40:41 smithi183 ceph-mon[25285]: from='client.14140 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi183", "addr": "172.21.15.183", "target": ["mon-mgr", ""]}]: dispatch 2024-03-20T08:40:42.454 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:40:42 smithi183 ceph-mon[25285]: Deploying cephadm binary to smithi183 2024-03-20T08:40:45.062 INFO:teuthology.orchestra.run.smithi183.stdout:/usr/bin/ceph: stdout Added host 'smithi183' with addr '172.21.15.183' 2024-03-20T08:40:45.063 INFO:teuthology.orchestra.run.smithi183.stdout:Deploying unmanaged mon service... 2024-03-20T08:40:45.666 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:40:45 smithi183 ceph-mon[25285]: from='mgr.14120 172.21.15.183:0/4190782725' entity='mgr.y' 2024-03-20T08:40:45.666 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:40:45 smithi183 ceph-mon[25285]: Added host smithi183 2024-03-20T08:40:45.666 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:40:45 smithi183 ceph-mon[25285]: from='mgr.14120 172.21.15.183:0/4190782725' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-20T08:40:46.230 INFO:teuthology.orchestra.run.smithi183.stdout:/usr/bin/ceph: stdout Scheduled mon update... 2024-03-20T08:40:46.230 INFO:teuthology.orchestra.run.smithi183.stdout:Deploying unmanaged mgr service... 2024-03-20T08:40:46.897 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:40:46 smithi183 ceph-mon[25285]: from='client.14142 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2024-03-20T08:40:46.897 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:40:46 smithi183 ceph-mon[25285]: Saving service mon spec with placement count:5 2024-03-20T08:40:46.898 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:40:46 smithi183 ceph-mon[25285]: from='mgr.14120 172.21.15.183:0/4190782725' entity='mgr.y' 2024-03-20T08:40:47.954 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:40:47 smithi183 ceph-mon[25285]: from='client.14144 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2024-03-20T08:40:47.954 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:40:47 smithi183 ceph-mon[25285]: Saving service mgr spec with placement count:2 2024-03-20T08:40:47.954 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:40:47 smithi183 ceph-mon[25285]: from='mgr.14120 172.21.15.183:0/4190782725' entity='mgr.y' 2024-03-20T08:40:48.342 INFO:teuthology.orchestra.run.smithi183.stdout:/usr/bin/ceph: stdout Scheduled mgr update... 2024-03-20T08:40:49.362 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:40:48 smithi183 ceph-mon[25285]: from='mgr.14120 172.21.15.183:0/4190782725' entity='mgr.y' 2024-03-20T08:40:49.362 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:40:48 smithi183 ceph-mon[25285]: from='mgr.14120 172.21.15.183:0/4190782725' entity='mgr.y' 2024-03-20T08:40:50.401 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:40:50 smithi183 ceph-mon[25285]: from='client.? 172.21.15.183:0/1524313607' entity='client.admin' 2024-03-20T08:40:51.164 INFO:teuthology.orchestra.run.smithi183.stdout:Enabling the dashboard module... 2024-03-20T08:40:51.454 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:40:51 smithi183 ceph-mon[25285]: from='client.? 172.21.15.183:0/3966074760' entity='client.admin' 2024-03-20T08:40:52.944 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:40:52 smithi183 ceph-mon[25285]: from='mgr.14120 172.21.15.183:0/4190782725' entity='mgr.y' 2024-03-20T08:40:52.945 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:40:52 smithi183 ceph-mon[25285]: from='mgr.14120 172.21.15.183:0/4190782725' entity='mgr.y' 2024-03-20T08:40:52.945 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:40:52 smithi183 ceph-mon[25285]: from='mgr.14120 172.21.15.183:0/4190782725' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi183", "name": "osd_memory_target"}]: dispatch 2024-03-20T08:40:52.945 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:40:52 smithi183 ceph-mon[25285]: from='mgr.14120 172.21.15.183:0/4190782725' entity='mgr.y' 2024-03-20T08:40:52.945 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:40:52 smithi183 ceph-mon[25285]: from='mgr.14120 172.21.15.183:0/4190782725' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.agent.smithi183", "caps": []}]: dispatch 2024-03-20T08:40:52.945 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:40:52 smithi183 ceph-mon[25285]: from='mgr.14120 172.21.15.183:0/4190782725' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.agent.smithi183", "caps": []}]': finished 2024-03-20T08:40:52.945 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:40:52 smithi183 ceph-mon[25285]: from='client.? 172.21.15.183:0/2163515517' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "dashboard"}]: dispatch 2024-03-20T08:40:52.945 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:40:52 smithi183 ceph-mon[25285]: Deploying daemon agent.smithi183 on smithi183 2024-03-20T08:40:52.945 INFO:journalctl@ceph.mgr.y.smithi183.stdout:Mar 20 08:40:52 smithi183 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-mgr-y[25513]: ignoring --setuser ceph since I am not root 2024-03-20T08:40:52.945 INFO:journalctl@ceph.mgr.y.smithi183.stdout:Mar 20 08:40:52 smithi183 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-mgr-y[25513]: ignoring --setgroup ceph since I am not root 2024-03-20T08:40:53.454 INFO:journalctl@ceph.mgr.y.smithi183.stdout:Mar 20 08:40:53 smithi183 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-mgr-y[25513]: 2024-03-20T08:40:53.203+0000 7fa385679200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-20T08:40:53.454 INFO:journalctl@ceph.mgr.y.smithi183.stdout:Mar 20 08:40:53 smithi183 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-mgr-y[25513]: 2024-03-20T08:40:53.301+0000 7fa385679200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-20T08:40:53.891 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:40:53 smithi183 ceph-mon[25285]: from='client.? 172.21.15.183:0/2163515517' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "dashboard"}]': finished 2024-03-20T08:40:53.891 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:40:53 smithi183 ceph-mon[25285]: mgrmap e9: y(active, since 19s) 2024-03-20T08:40:54.184 INFO:journalctl@ceph.mgr.y.smithi183.stdout:Mar 20 08:40:53 smithi183 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-mgr-y[25513]: 2024-03-20T08:40:53.958+0000 7fa385679200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-20T08:40:54.184 INFO:journalctl@ceph.mgr.y.smithi183.stdout:Mar 20 08:40:54 smithi183 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-mgr-y[25513]: 2024-03-20T08:40:54.048+0000 7fa385679200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-20T08:40:54.454 INFO:journalctl@ceph.mgr.y.smithi183.stdout:Mar 20 08:40:54 smithi183 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-mgr-y[25513]: 2024-03-20T08:40:54.191+0000 7fa385679200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-20T08:40:54.454 INFO:journalctl@ceph.mgr.y.smithi183.stdout:Mar 20 08:40:54 smithi183 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-mgr-y[25513]: 2024-03-20T08:40:54.286+0000 7fa385679200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-20T08:40:54.454 INFO:journalctl@ceph.mgr.y.smithi183.stdout:Mar 20 08:40:54 smithi183 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-mgr-y[25513]: 2024-03-20T08:40:54.369+0000 7fa385679200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-20T08:40:54.471 INFO:teuthology.orchestra.run.smithi183.stdout:/usr/bin/ceph: stdout { 2024-03-20T08:40:54.471 INFO:teuthology.orchestra.run.smithi183.stdout:/usr/bin/ceph: stdout "epoch": 9, 2024-03-20T08:40:54.471 INFO:teuthology.orchestra.run.smithi183.stdout:/usr/bin/ceph: stdout "available": true, 2024-03-20T08:40:54.471 INFO:teuthology.orchestra.run.smithi183.stdout:/usr/bin/ceph: stdout "active_name": "y", 2024-03-20T08:40:54.471 INFO:teuthology.orchestra.run.smithi183.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-03-20T08:40:54.471 INFO:teuthology.orchestra.run.smithi183.stdout:/usr/bin/ceph: stdout } 2024-03-20T08:40:54.471 INFO:teuthology.orchestra.run.smithi183.stdout:Waiting for the mgr to restart... 2024-03-20T08:40:54.471 INFO:teuthology.orchestra.run.smithi183.stdout:Waiting for mgr epoch 9... 2024-03-20T08:40:54.727 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:40:54 smithi183 ceph-mon[25285]: from='client.? 172.21.15.183:0/3947935011' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-03-20T08:40:54.727 INFO:journalctl@ceph.mgr.y.smithi183.stdout:Mar 20 08:40:54 smithi183 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-mgr-y[25513]: 2024-03-20T08:40:54.640+0000 7fa385679200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-20T08:40:55.115 INFO:journalctl@ceph.mgr.y.smithi183.stdout:Mar 20 08:40:54 smithi183 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-mgr-y[25513]: 2024-03-20T08:40:54.751+0000 7fa385679200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-20T08:40:55.115 INFO:journalctl@ceph.mgr.y.smithi183.stdout:Mar 20 08:40:54 smithi183 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-mgr-y[25513]: 2024-03-20T08:40:54.837+0000 7fa385679200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-20T08:40:55.115 INFO:journalctl@ceph.mgr.y.smithi183.stdout:Mar 20 08:40:54 smithi183 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-mgr-y[25513]: 2024-03-20T08:40:54.936+0000 7fa385679200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-20T08:40:55.454 INFO:journalctl@ceph.mgr.y.smithi183.stdout:Mar 20 08:40:55 smithi183 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-mgr-y[25513]: 2024-03-20T08:40:55.115+0000 7fa385679200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-20T08:40:56.454 INFO:journalctl@ceph.mgr.y.smithi183.stdout:Mar 20 08:40:56 smithi183 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-mgr-y[25513]: 2024-03-20T08:40:56.075+0000 7fa385679200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-20T08:40:56.454 INFO:journalctl@ceph.mgr.y.smithi183.stdout:Mar 20 08:40:56 smithi183 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-mgr-y[25513]: 2024-03-20T08:40:56.205+0000 7fa385679200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-20T08:40:56.828 INFO:journalctl@ceph.mgr.y.smithi183.stdout:Mar 20 08:40:56 smithi183 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-mgr-y[25513]: 2024-03-20T08:40:56.542+0000 7fa385679200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-20T08:40:56.829 INFO:journalctl@ceph.mgr.y.smithi183.stdout:Mar 20 08:40:56 smithi183 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-mgr-y[25513]: 2024-03-20T08:40:56.631+0000 7fa385679200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-20T08:40:56.829 INFO:journalctl@ceph.mgr.y.smithi183.stdout:Mar 20 08:40:56 smithi183 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-mgr-y[25513]: 2024-03-20T08:40:56.828+0000 7fa385679200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-20T08:40:57.204 INFO:journalctl@ceph.mgr.y.smithi183.stdout:Mar 20 08:40:57 smithi183 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-mgr-y[25513]: 2024-03-20T08:40:57.013+0000 7fa385679200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-20T08:40:57.561 INFO:journalctl@ceph.mgr.y.smithi183.stdout:Mar 20 08:40:57 smithi183 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-mgr-y[25513]: 2024-03-20T08:40:57.300+0000 7fa385679200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-20T08:40:57.561 INFO:journalctl@ceph.mgr.y.smithi183.stdout:Mar 20 08:40:57 smithi183 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-mgr-y[25513]: 2024-03-20T08:40:57.560+0000 7fa385679200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-20T08:40:57.954 INFO:journalctl@ceph.mgr.y.smithi183.stdout:Mar 20 08:40:57 smithi183 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-mgr-y[25513]: 2024-03-20T08:40:57.798+0000 7fa385679200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-20T08:40:58.954 INFO:journalctl@ceph.mgr.y.smithi183.stdout:Mar 20 08:40:58 smithi183 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-mgr-y[25513]: 2024-03-20T08:40:58.601+0000 7fa385679200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-20T08:40:59.454 INFO:journalctl@ceph.mgr.y.smithi183.stdout:Mar 20 08:40:59 smithi183 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-mgr-y[25513]: 2024-03-20T08:40:59.170+0000 7fa385679200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-20T08:40:59.454 INFO:journalctl@ceph.mgr.y.smithi183.stdout:Mar 20 08:40:59 smithi183 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-mgr-y[25513]: 2024-03-20T08:40:59.269+0000 7fa385679200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-20T08:40:59.954 INFO:journalctl@ceph.mgr.y.smithi183.stdout:Mar 20 08:40:59 smithi183 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-mgr-y[25513]: 2024-03-20T08:40:59.487+0000 7fa385679200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-20T08:40:59.954 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:40:59 smithi183 ceph-mon[25285]: Active manager daemon y restarted 2024-03-20T08:40:59.954 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:40:59 smithi183 ceph-mon[25285]: Activating manager daemon y 2024-03-20T08:40:59.954 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:40:59 smithi183 ceph-mon[25285]: osdmap e3: 0 total, 0 up, 0 in 2024-03-20T08:40:59.955 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:40:59 smithi183 ceph-mon[25285]: mgrmap e10: y(active, starting, since 0.00322568s) 2024-03-20T08:40:59.955 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:40:59 smithi183 ceph-mon[25285]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-20T08:40:59.955 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:40:59 smithi183 ceph-mon[25285]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-20T08:40:59.955 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:40:59 smithi183 ceph-mon[25285]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-20T08:40:59.955 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:40:59 smithi183 ceph-mon[25285]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-20T08:40:59.955 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:40:59 smithi183 ceph-mon[25285]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-20T08:40:59.955 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:40:59 smithi183 ceph-mon[25285]: Manager daemon y is now available 2024-03-20T08:40:59.955 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:40:59 smithi183 ceph-mon[25285]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:40:59.955 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:40:59 smithi183 ceph-mon[25285]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-20T08:40:59.955 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:40:59 smithi183 ceph-mon[25285]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-20T08:41:00.704 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:41:00 smithi183 ceph-mon[25285]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-20T08:41:00.704 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:41:00 smithi183 ceph-mon[25285]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-20T08:41:00.704 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:41:00 smithi183 ceph-mon[25285]: mgrmap e11: y(active, since 1.00642s) 2024-03-20T08:41:01.447 INFO:teuthology.orchestra.run.smithi183.stdout:/usr/bin/ceph: stdout { 2024-03-20T08:41:01.447 INFO:teuthology.orchestra.run.smithi183.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 11, 2024-03-20T08:41:01.447 INFO:teuthology.orchestra.run.smithi183.stdout:/usr/bin/ceph: stdout "initialized": true 2024-03-20T08:41:01.447 INFO:teuthology.orchestra.run.smithi183.stdout:/usr/bin/ceph: stdout } 2024-03-20T08:41:01.447 INFO:teuthology.orchestra.run.smithi183.stdout:mgr epoch 9 is available 2024-03-20T08:41:01.447 INFO:teuthology.orchestra.run.smithi183.stdout:Generating a dashboard self-signed certificate... 2024-03-20T08:41:01.855 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:41:01 smithi183 ceph-mon[25285]: from='client.14156 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-03-20T08:41:01.855 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:41:01 smithi183 ceph-mon[25285]: from='client.14156 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-03-20T08:41:01.855 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:41:01 smithi183 ceph-mon[25285]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:41:01.855 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:41:01 smithi183 ceph-mon[25285]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi183", "name": "osd_memory_target"}]: dispatch 2024-03-20T08:41:01.855 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:41:01 smithi183 ceph-mon[25285]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:41:01.855 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:41:01 smithi183 ceph-mon[25285]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.agent.smithi183", "caps": []}]: dispatch 2024-03-20T08:41:02.761 INFO:teuthology.orchestra.run.smithi183.stdout:/usr/bin/ceph: stdout Self-signed certificate created 2024-03-20T08:41:02.761 INFO:teuthology.orchestra.run.smithi183.stdout:Creating initial admin user... 2024-03-20T08:41:02.867 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:41:02 smithi183 ceph-mon[25285]: [20/Mar/2024:08:41:01] ENGINE Bus STARTING 2024-03-20T08:41:02.868 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:41:02 smithi183 ceph-mon[25285]: [20/Mar/2024:08:41:01] ENGINE Serving on http://172.21.15.183:8765 2024-03-20T08:41:02.868 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:41:02 smithi183 ceph-mon[25285]: [20/Mar/2024:08:41:01] ENGINE Serving on https://172.21.15.183:7150 2024-03-20T08:41:02.868 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:41:02 smithi183 ceph-mon[25285]: [20/Mar/2024:08:41:01] ENGINE Bus STARTED 2024-03-20T08:41:02.868 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:41:02 smithi183 ceph-mon[25285]: Deploying daemon agent.smithi183 on smithi183 2024-03-20T08:41:02.868 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:41:02 smithi183 ceph-mon[25285]: mgrmap e12: y(active, since 2s) 2024-03-20T08:41:02.868 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:41:02 smithi183 ceph-mon[25285]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:41:02.868 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:41:02 smithi183 ceph-mon[25285]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:41:03.931 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:41:03 smithi183 ceph-mon[25285]: from='client.14164 -' entity='client.admin' cmd=[{"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}]: dispatch 2024-03-20T08:41:04.193 INFO:teuthology.orchestra.run.smithi183.stdout:/usr/bin/ceph: stdout {"username": "admin", "password": "$2b$12$nAnSIOdh/tpXo3YbLghNEuEcQfPxC77gBC0uHVxl2eEE8uauNAvM6", "roles": ["administrator"], "name": null, "email": null, "lastUpdate": 1710924063, "enabled": true, "pwdExpirationDate": null, "pwdUpdateRequired": true} 2024-03-20T08:41:04.193 INFO:teuthology.orchestra.run.smithi183.stdout:Fetching dashboard port number... 2024-03-20T08:41:04.791 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:41:04 smithi183 ceph-mon[25285]: from='client.14166 -' entity='client.admin' cmd=[{"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}]: dispatch 2024-03-20T08:41:04.791 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:41:04 smithi183 ceph-mon[25285]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:41:05.374 INFO:teuthology.orchestra.run.smithi183.stdout:/usr/bin/ceph: stdout 8443 2024-03-20T08:41:05.384 INFO:teuthology.orchestra.run.smithi183.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-03-20T08:41:05.384 INFO:teuthology.orchestra.run.smithi183.stdout:systemctl: stdout disabled 2024-03-20T08:41:05.391 INFO:teuthology.orchestra.run.smithi183.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-03-20T08:41:05.392 INFO:teuthology.orchestra.run.smithi183.stdout:systemctl: stdout inactive 2024-03-20T08:41:05.392 INFO:teuthology.orchestra.run.smithi183.stdout:firewalld.service is not enabled 2024-03-20T08:41:05.392 INFO:teuthology.orchestra.run.smithi183.stdout:Not possible to open ports <[8443]>. firewalld.service is not available 2024-03-20T08:41:05.394 INFO:teuthology.orchestra.run.smithi183.stdout:Ceph Dashboard is now available at: 2024-03-20T08:41:05.394 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-20T08:41:05.394 INFO:teuthology.orchestra.run.smithi183.stdout: URL: https://smithi183.front.sepia.ceph.com:8443/ 2024-03-20T08:41:05.394 INFO:teuthology.orchestra.run.smithi183.stdout: User: admin 2024-03-20T08:41:05.394 INFO:teuthology.orchestra.run.smithi183.stdout: Password: sttui3ru27 2024-03-20T08:41:05.394 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-20T08:41:05.395 INFO:teuthology.orchestra.run.smithi183.stdout:Saving cluster configuration to /var/lib/ceph/4c3f3eba-e695-11ee-95c9-87774f69a715/config directory 2024-03-20T08:41:05.395 INFO:teuthology.orchestra.run.smithi183.stdout:Enabling autotune for osd_memory_target 2024-03-20T08:41:05.954 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:41:05 smithi183 ceph-mon[25285]: from='client.? 172.21.15.183:0/2016141743' entity='client.admin' cmd=[{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]: dispatch 2024-03-20T08:41:06.686 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:41:06 smithi183 ceph-mon[25285]: mgrmap e13: y(active, since 6s) 2024-03-20T08:41:08.078 INFO:teuthology.orchestra.run.smithi183.stdout:/usr/bin/ceph: stderr set mgr/dashboard/cluster/status 2024-03-20T08:41:08.078 INFO:teuthology.orchestra.run.smithi183.stdout:You can access the Ceph CLI as following in case of multi-cluster or non-default config: 2024-03-20T08:41:08.078 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-20T08:41:08.078 INFO:teuthology.orchestra.run.smithi183.stdout: sudo /home/ubuntu/cephtest/cephadm shell --fsid 4c3f3eba-e695-11ee-95c9-87774f69a715 -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring 2024-03-20T08:41:08.078 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-20T08:41:08.078 INFO:teuthology.orchestra.run.smithi183.stdout:Or, if you are only running a single cluster on this host: 2024-03-20T08:41:08.078 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-20T08:41:08.079 INFO:teuthology.orchestra.run.smithi183.stdout: sudo /home/ubuntu/cephtest/cephadm shell 2024-03-20T08:41:08.079 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-20T08:41:08.079 INFO:teuthology.orchestra.run.smithi183.stdout:Please consider enabling telemetry to help improve Ceph: 2024-03-20T08:41:08.079 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-20T08:41:08.079 INFO:teuthology.orchestra.run.smithi183.stdout: ceph telemetry on 2024-03-20T08:41:08.079 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-20T08:41:08.079 INFO:teuthology.orchestra.run.smithi183.stdout:For more information see: 2024-03-20T08:41:08.079 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-20T08:41:08.079 INFO:teuthology.orchestra.run.smithi183.stdout: https://docs.ceph.com/en/latest/mgr/telemetry/ 2024-03-20T08:41:08.079 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-20T08:41:08.079 INFO:teuthology.orchestra.run.smithi183.stdout:Bootstrap complete. 2024-03-20T08:41:08.124 INFO:tasks.cephadm:Fetching config... 2024-03-20T08:41:08.124 DEBUG:teuthology.orchestra.run.smithi183:> set -ex 2024-03-20T08:41:08.124 DEBUG:teuthology.orchestra.run.smithi183:> dd if=/etc/ceph/ceph.conf of=/dev/stdout 2024-03-20T08:41:08.139 INFO:tasks.cephadm:Fetching client.admin keyring... 2024-03-20T08:41:08.139 DEBUG:teuthology.orchestra.run.smithi183:> set -ex 2024-03-20T08:41:08.139 DEBUG:teuthology.orchestra.run.smithi183:> dd if=/etc/ceph/ceph.client.admin.keyring of=/dev/stdout 2024-03-20T08:41:08.195 INFO:tasks.cephadm:Fetching mon keyring... 2024-03-20T08:41:08.195 DEBUG:teuthology.orchestra.run.smithi183:> set -ex 2024-03-20T08:41:08.195 DEBUG:teuthology.orchestra.run.smithi183:> sudo dd if=/var/lib/ceph/4c3f3eba-e695-11ee-95c9-87774f69a715/mon.a/keyring of=/dev/stdout 2024-03-20T08:41:08.362 INFO:tasks.cephadm:Fetching pub ssh key... 2024-03-20T08:41:08.362 DEBUG:teuthology.orchestra.run.smithi183:> set -ex 2024-03-20T08:41:08.362 DEBUG:teuthology.orchestra.run.smithi183:> dd if=/home/ubuntu/cephtest/ceph.pub of=/dev/stdout 2024-03-20T08:41:08.417 INFO:tasks.cephadm:Installing pub ssh key for root users... 2024-03-20T08:41:08.418 DEBUG:teuthology.orchestra.run.smithi183:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCu7rXZCzuPfK508y3lsFrV1JQq4dlVbN73hCrA76eus1RG6/erFzIM9L71J2LAa+HceWm7jIhwtGTJlzGuJgG4fgDirQNTb+w2hU0XDLBmwr1pNO5f2BdvPpi7t26qUwhLs6P/zmEsyyoxL/iqelDcR/vb/FUq5r6UF/TIFGKwKkiuNRyREhquxGP7gXuMevi/PKeM+bDoYmoCfZ3TGGjgrfi65YifYEomcxsdzQJBB2BLChkOkwNQBMsEKYZo4RhBgpcGJA7IHk7cH+lRZxDFLzglEFEd5aqSTw97TQAgjwnD4bw03mIMTMw+RKoxmZWjCMLqTfYjze1SiJOIA4dqplIXHiUMB8evBa0b7j5APr0O47iwz1jdvuTecipAN0vxxTh0scdoUEk7lgOhE+eb2qPyrs0lxSiirkeFJDOMDpmiFJmb01/VHEgwrgJT1W21FE17O8Y8Z/E9M2U7A7r6XlLtQyIZlD7oPzVcnNdweatZWvo6cma8tvvdZPaMI98= ceph-4c3f3eba-e695-11ee-95c9-87774f69a715' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-03-20T08:41:08.575 INFO:teuthology.orchestra.run.smithi183.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCu7rXZCzuPfK508y3lsFrV1JQq4dlVbN73hCrA76eus1RG6/erFzIM9L71J2LAa+HceWm7jIhwtGTJlzGuJgG4fgDirQNTb+w2hU0XDLBmwr1pNO5f2BdvPpi7t26qUwhLs6P/zmEsyyoxL/iqelDcR/vb/FUq5r6UF/TIFGKwKkiuNRyREhquxGP7gXuMevi/PKeM+bDoYmoCfZ3TGGjgrfi65YifYEomcxsdzQJBB2BLChkOkwNQBMsEKYZo4RhBgpcGJA7IHk7cH+lRZxDFLzglEFEd5aqSTw97TQAgjwnD4bw03mIMTMw+RKoxmZWjCMLqTfYjze1SiJOIA4dqplIXHiUMB8evBa0b7j5APr0O47iwz1jdvuTecipAN0vxxTh0scdoUEk7lgOhE+eb2qPyrs0lxSiirkeFJDOMDpmiFJmb01/VHEgwrgJT1W21FE17O8Y8Z/E9M2U7A7r6XlLtQyIZlD7oPzVcnNdweatZWvo6cma8tvvdZPaMI98= ceph-4c3f3eba-e695-11ee-95c9-87774f69a715 2024-03-20T08:41:08.590 DEBUG:teuthology.orchestra.run.smithi194:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCu7rXZCzuPfK508y3lsFrV1JQq4dlVbN73hCrA76eus1RG6/erFzIM9L71J2LAa+HceWm7jIhwtGTJlzGuJgG4fgDirQNTb+w2hU0XDLBmwr1pNO5f2BdvPpi7t26qUwhLs6P/zmEsyyoxL/iqelDcR/vb/FUq5r6UF/TIFGKwKkiuNRyREhquxGP7gXuMevi/PKeM+bDoYmoCfZ3TGGjgrfi65YifYEomcxsdzQJBB2BLChkOkwNQBMsEKYZo4RhBgpcGJA7IHk7cH+lRZxDFLzglEFEd5aqSTw97TQAgjwnD4bw03mIMTMw+RKoxmZWjCMLqTfYjze1SiJOIA4dqplIXHiUMB8evBa0b7j5APr0O47iwz1jdvuTecipAN0vxxTh0scdoUEk7lgOhE+eb2qPyrs0lxSiirkeFJDOMDpmiFJmb01/VHEgwrgJT1W21FE17O8Y8Z/E9M2U7A7r6XlLtQyIZlD7oPzVcnNdweatZWvo6cma8tvvdZPaMI98= ceph-4c3f3eba-e695-11ee-95c9-87774f69a715' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-03-20T08:41:08.631 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:41:08 smithi183 ceph-mon[25285]: from='client.? 172.21.15.183:0/2957430682' entity='client.admin' 2024-03-20T08:41:08.637 INFO:teuthology.orchestra.run.smithi194.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCu7rXZCzuPfK508y3lsFrV1JQq4dlVbN73hCrA76eus1RG6/erFzIM9L71J2LAa+HceWm7jIhwtGTJlzGuJgG4fgDirQNTb+w2hU0XDLBmwr1pNO5f2BdvPpi7t26qUwhLs6P/zmEsyyoxL/iqelDcR/vb/FUq5r6UF/TIFGKwKkiuNRyREhquxGP7gXuMevi/PKeM+bDoYmoCfZ3TGGjgrfi65YifYEomcxsdzQJBB2BLChkOkwNQBMsEKYZo4RhBgpcGJA7IHk7cH+lRZxDFLzglEFEd5aqSTw97TQAgjwnD4bw03mIMTMw+RKoxmZWjCMLqTfYjze1SiJOIA4dqplIXHiUMB8evBa0b7j5APr0O47iwz1jdvuTecipAN0vxxTh0scdoUEk7lgOhE+eb2qPyrs0lxSiirkeFJDOMDpmiFJmb01/VHEgwrgJT1W21FE17O8Y8Z/E9M2U7A7r6XlLtQyIZlD7oPzVcnNdweatZWvo6cma8tvvdZPaMI98= ceph-4c3f3eba-e695-11ee-95c9-87774f69a715 2024-03-20T08:41:08.652 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4c3f3eba-e695-11ee-95c9-87774f69a715 -- ceph config set mgr mgr/cephadm/allow_ptrace true 2024-03-20T08:41:08.943 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/4c3f3eba-e695-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T08:41:09.925 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:41:09 smithi183 ceph-mon[25285]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:41:09.925 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:41:09 smithi183 ceph-mon[25285]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:41:09.925 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:41:09 smithi183 ceph-mon[25285]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:41:09.925 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:41:09 smithi183 ceph-mon[25285]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-20T08:41:09.926 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:41:09 smithi183 ceph-mon[25285]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:41:09.926 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:41:09 smithi183 ceph-mon[25285]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:41:09.926 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:41:09 smithi183 ceph-mon[25285]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:41:09.926 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:41:09 smithi183 ceph-mon[25285]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:41:09.926 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:41:09 smithi183 ceph-mon[25285]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:41:09.926 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:41:09 smithi183 ceph-mon[25285]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-20T08:41:09.926 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:41:09 smithi183 ceph-mon[25285]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:41:09.926 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:41:09 smithi183 ceph-mon[25285]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:41:10.954 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:41:10 smithi183 ceph-mon[25285]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:41:10.954 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:41:10 smithi183 ceph-mon[25285]: from='client.? 172.21.15.183:0/1589892912' entity='client.admin' 2024-03-20T08:41:10.954 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:41:10 smithi183 ceph-mon[25285]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-20T08:41:10.954 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:41:10 smithi183 ceph-mon[25285]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:41:11.508 INFO:tasks.cephadm:Distributing conf and client.admin keyring to all hosts + 0755 2024-03-20T08:41:11.509 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4c3f3eba-e695-11ee-95c9-87774f69a715 -- ceph orch client-keyring set client.admin '*' --mode 0755 2024-03-20T08:41:12.568 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/4c3f3eba-e695-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T08:41:14.558 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:41:14 smithi183 ceph-mon[25285]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:41:14.558 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:41:14 smithi183 ceph-mon[25285]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:41:16.204 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:41:16 smithi183 ceph-mon[25285]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:41:16.204 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:41:16 smithi183 ceph-mon[25285]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:41:16.204 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:41:16 smithi183 ceph-mon[25285]: from='client.14176 -' entity='client.admin' cmd=[{"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}]: dispatch 2024-03-20T08:41:16.204 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:41:16 smithi183 ceph-mon[25285]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:41:16.204 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:41:16 smithi183 ceph-mon[25285]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-20T08:41:16.204 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:41:16 smithi183 ceph-mon[25285]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-20T08:41:16.205 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:41:16 smithi183 ceph-mon[25285]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-20T08:41:16.205 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:41:16 smithi183 ceph-mon[25285]: Updating smithi183:/etc/ceph/ceph.conf 2024-03-20T08:41:16.299 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi194 2024-03-20T08:41:16.299 DEBUG:teuthology.orchestra.run.smithi194:> set -ex 2024-03-20T08:41:16.299 DEBUG:teuthology.orchestra.run.smithi194:> dd of=/etc/ceph/ceph.conf 2024-03-20T08:41:16.318 DEBUG:teuthology.orchestra.run.smithi194:> set -ex 2024-03-20T08:41:16.318 DEBUG:teuthology.orchestra.run.smithi194:> dd of=/etc/ceph/ceph.client.admin.keyring 2024-03-20T08:41:16.375 INFO:tasks.cephadm:Adding host smithi194 to orchestrator... 2024-03-20T08:41:16.375 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4c3f3eba-e695-11ee-95c9-87774f69a715 -- ceph orch host add smithi194 2024-03-20T08:41:16.611 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/4c3f3eba-e695-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T08:41:17.693 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:41:17 smithi183 ceph-mon[25285]: Updating smithi183:/var/lib/ceph/4c3f3eba-e695-11ee-95c9-87774f69a715/config/ceph.conf 2024-03-20T08:41:17.693 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:41:17 smithi183 ceph-mon[25285]: Updating smithi183:/etc/ceph/ceph.client.admin.keyring 2024-03-20T08:41:17.693 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:41:17 smithi183 ceph-mon[25285]: Updating smithi183:/var/lib/ceph/4c3f3eba-e695-11ee-95c9-87774f69a715/config/ceph.client.admin.keyring 2024-03-20T08:41:17.693 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:41:17 smithi183 ceph-mon[25285]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:41:17.693 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:41:17 smithi183 ceph-mon[25285]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:41:17.693 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:41:17 smithi183 ceph-mon[25285]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:41:19.204 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:41:18 smithi183 ceph-mon[25285]: from='client.14178 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi194", "target": ["mon-mgr", ""]}]: dispatch 2024-03-20T08:41:20.954 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:41:20 smithi183 ceph-mon[25285]: Deploying cephadm binary to smithi194 2024-03-20T08:41:20.954 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:41:20 smithi183 ceph-mon[25285]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-20T08:41:22.873 INFO:teuthology.orchestra.run.smithi183.stdout:Added host 'smithi194' with addr '172.21.15.194' 2024-03-20T08:41:22.873 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:41:22 smithi183 ceph-mon[25285]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-20T08:41:23.507 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4c3f3eba-e695-11ee-95c9-87774f69a715 -- ceph orch host ls --format=json 2024-03-20T08:41:23.745 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/4c3f3eba-e695-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T08:41:24.126 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:41:23 smithi183 ceph-mon[25285]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:41:24.126 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:41:23 smithi183 ceph-mon[25285]: Added host smithi194 2024-03-20T08:41:24.126 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:41:23 smithi183 ceph-mon[25285]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-20T08:41:24.127 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:41:23 smithi183 ceph-mon[25285]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:41:24.127 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:41:23 smithi183 ceph-mon[25285]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-20T08:41:25.132 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:41:24 smithi183 ceph-mon[25285]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:41:25.132 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:41:24 smithi183 ceph-mon[25285]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:41:25.132 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:41:24 smithi183 ceph-mon[25285]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi194", "name": "osd_memory_target"}]: dispatch 2024-03-20T08:41:25.133 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:41:24 smithi183 ceph-mon[25285]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-20T08:41:25.133 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:41:24 smithi183 ceph-mon[25285]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-20T08:41:25.133 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:41:24 smithi183 ceph-mon[25285]: Updating smithi194:/etc/ceph/ceph.conf 2024-03-20T08:41:25.133 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:41:24 smithi183 ceph-mon[25285]: Updating smithi194:/var/lib/ceph/4c3f3eba-e695-11ee-95c9-87774f69a715/config/ceph.conf 2024-03-20T08:41:25.133 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:41:24 smithi183 ceph-mon[25285]: Updating smithi194:/etc/ceph/ceph.client.admin.keyring 2024-03-20T08:41:25.133 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:41:24 smithi183 ceph-mon[25285]: Updating smithi194:/var/lib/ceph/4c3f3eba-e695-11ee-95c9-87774f69a715/config/ceph.client.admin.keyring 2024-03-20T08:41:25.133 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:41:24 smithi183 ceph-mon[25285]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:41:25.133 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:41:24 smithi183 ceph-mon[25285]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:41:25.133 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:41:24 smithi183 ceph-mon[25285]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.agent.smithi194", "caps": []}]: dispatch 2024-03-20T08:41:25.133 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:41:24 smithi183 ceph-mon[25285]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.agent.smithi194", "caps": []}]': finished 2024-03-20T08:41:25.578 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-20T08:41:25.578 INFO:teuthology.orchestra.run.smithi183.stdout:[{"addr": "172.21.15.183", "hostname": "smithi183", "labels": [], "status": ""}, {"addr": "172.21.15.194", "hostname": "smithi194", "labels": [], "status": ""}] 2024-03-20T08:41:26.170 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:41:25 smithi183 ceph-mon[25285]: Deploying daemon agent.smithi194 on smithi194 2024-03-20T08:41:26.170 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:41:25 smithi183 ceph-mon[25285]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-20T08:41:26.171 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:41:25 smithi183 ceph-mon[25285]: from='client.14180 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-20T08:41:26.171 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:41:25 smithi183 ceph-mon[25285]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:41:26.171 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:41:25 smithi183 ceph-mon[25285]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:41:26.171 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:41:25 smithi183 ceph-mon[25285]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:41:26.171 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:41:25 smithi183 ceph-mon[25285]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-20T08:41:26.171 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:41:25 smithi183 ceph-mon[25285]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-20T08:41:26.171 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:41:25 smithi183 ceph-mon[25285]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-20T08:41:26.171 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:41:25 smithi183 ceph-mon[25285]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:41:26.172 INFO:tasks.cephadm:Setting crush tunables to default 2024-03-20T08:41:26.172 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4c3f3eba-e695-11ee-95c9-87774f69a715 -- ceph osd crush tunables default 2024-03-20T08:41:26.405 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/4c3f3eba-e695-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T08:41:27.185 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:41:26 smithi183 ceph-mon[25285]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-20T08:41:27.185 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:41:26 smithi183 ceph-mon[25285]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-20T08:41:27.186 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:41:26 smithi183 ceph-mon[25285]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-20T08:41:27.186 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:41:26 smithi183 ceph-mon[25285]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:41:27.186 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:41:26 smithi183 ceph-mon[25285]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:41:27.186 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:41:26 smithi183 ceph-mon[25285]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:41:27.186 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:41:26 smithi183 ceph-mon[25285]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:41:27.954 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:41:27 smithi183 ceph-mon[25285]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-20T08:41:28.926 INFO:teuthology.orchestra.run.smithi183.stderr:adjusted tunables profile to default 2024-03-20T08:41:29.204 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:41:28 smithi183 ceph-mon[25285]: from='client.? 172.21.15.183:0/21530245' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-03-20T08:41:29.501 INFO:tasks.cephadm:Adding mon.a on smithi183 2024-03-20T08:41:29.501 INFO:tasks.cephadm:Adding mon.c on smithi183 2024-03-20T08:41:29.502 INFO:tasks.cephadm:Adding mon.b on smithi194 2024-03-20T08:41:29.502 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4c3f3eba-e695-11ee-95c9-87774f69a715 -- ceph orch apply mon '3;smithi183:172.21.15.183=a;smithi183:[v2:172.21.15.183:3301,v1:172.21.15.183:6790]=c;smithi194:172.21.15.194=b' 2024-03-20T08:41:29.753 INFO:teuthology.orchestra.run.smithi194.stderr:Inferring config /var/lib/ceph/4c3f3eba-e695-11ee-95c9-87774f69a715/config/ceph.conf 2024-03-20T08:41:30.202 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:41:29 smithi183 ceph-mon[25285]: from='client.? 172.21.15.183:0/21530245' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-03-20T08:41:30.202 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:41:29 smithi183 ceph-mon[25285]: osdmap e4: 0 total, 0 up, 0 in 2024-03-20T08:41:30.202 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:41:29 smithi183 ceph-mon[25285]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:41:30.203 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:41:29 smithi183 ceph-mon[25285]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:41:30.203 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:41:29 smithi183 ceph-mon[25285]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-20T08:41:30.203 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:41:29 smithi183 ceph-mon[25285]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:41:31.563 INFO:teuthology.orchestra.run.smithi194.stdout:Scheduled mon update... 2024-03-20T08:41:32.120 DEBUG:teuthology.orchestra.run.smithi183:mon.c> sudo journalctl -f -n 0 -u ceph-4c3f3eba-e695-11ee-95c9-87774f69a715@mon.c.service 2024-03-20T08:41:32.123 DEBUG:teuthology.orchestra.run.smithi194:mon.b> sudo journalctl -f -n 0 -u ceph-4c3f3eba-e695-11ee-95c9-87774f69a715@mon.b.service 2024-03-20T08:41:32.126 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-03-20T08:41:32.126 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4c3f3eba-e695-11ee-95c9-87774f69a715 -- ceph mon dump -f json 2024-03-20T08:41:32.149 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:41:31 smithi183 ceph-mon[25285]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:41:32.149 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:41:31 smithi183 ceph-mon[25285]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:41:32.149 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:41:31 smithi183 ceph-mon[25285]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:41:32.150 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:41:31 smithi183 ceph-mon[25285]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-20T08:41:32.150 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:41:31 smithi183 ceph-mon[25285]: from='client.14184 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi183:172.21.15.183=a;smithi183:[v2:172.21.15.183:3301,v1:172.21.15.183:6790]=c;smithi194:172.21.15.194=b", "target": ["mon-mgr", ""]}]: dispatch 2024-03-20T08:41:32.150 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:41:31 smithi183 ceph-mon[25285]: Saving service mon spec with placement smithi183:172.21.15.183=a;smithi183:[v2:172.21.15.183:3301,v1:172.21.15.183:6790]=c;smithi194:172.21.15.194=b;count:3 2024-03-20T08:41:32.150 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:41:31 smithi183 ceph-mon[25285]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:41:32.150 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:41:31 smithi183 ceph-mon[25285]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-20T08:41:32.150 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:41:31 smithi183 ceph-mon[25285]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-20T08:41:32.150 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:41:31 smithi183 ceph-mon[25285]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-20T08:41:32.150 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:41:31 smithi183 ceph-mon[25285]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:41:32.150 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:41:31 smithi183 ceph-mon[25285]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-20T08:41:32.150 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:41:31 smithi183 ceph-mon[25285]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-20T08:41:32.150 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:41:31 smithi183 ceph-mon[25285]: Deploying daemon mon.b on smithi194 2024-03-20T08:41:33.690 INFO:teuthology.orchestra.run.smithi194.stderr:Inferring config /var/lib/ceph/4c3f3eba-e695-11ee-95c9-87774f69a715/config/ceph.conf 2024-03-20T08:41:34.954 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:41:34 smithi183 ceph-mon[25285]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-20T08:41:35.362 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:41:35 smithi194 systemd[1]: Starting Ceph mon.b for 4c3f3eba-e695-11ee-95c9-87774f69a715... 2024-03-20T08:41:35.638 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:41:35 smithi194 podman[27189]: 2024-03-20 08:41:35.465824372 +0000 UTC m=+0.020248286 image pull 9ceba81fb2c8d8682cb159527017d591637b153b3aa4a390ffa8586db6dcda00 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:976a8b2d06e3c6bfd12488297e8584dd704cc06c3e976c7e0f718c13e238e912 2024-03-20T08:41:35.958 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:41:35 smithi194 podman[27189]: 2024-03-20 08:41:35.773577548 +0000 UTC m=+0.328001444 container create da6a20d027a94b6237e2f032b342d32e508375805cb7e427ff891e7d6919fe6a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:976a8b2d06e3c6bfd12488297e8584dd704cc06c3e976c7e0f718c13e238e912, name=ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-mon-b, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240307, ceph=True, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, RELEASE=wip-yuri5-testing-2024-03-18-1144-e7e80a6, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-20T08:41:36.389 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:41:36 smithi194 podman[27189]: 2024-03-20 08:41:36.126547208 +0000 UTC m=+0.680971104 container init da6a20d027a94b6237e2f032b342d32e508375805cb7e427ff891e7d6919fe6a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:976a8b2d06e3c6bfd12488297e8584dd704cc06c3e976c7e0f718c13e238e912, name=ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-mon-b, RELEASE=wip-yuri5-testing-2024-03-18-1144-e7e80a6, GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.build-date=20240307, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-20T08:41:36.390 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:41:36 smithi194 podman[27189]: 2024-03-20 08:41:36.130578337 +0000 UTC m=+0.685002236 container start da6a20d027a94b6237e2f032b342d32e508375805cb7e427ff891e7d6919fe6a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:976a8b2d06e3c6bfd12488297e8584dd704cc06c3e976c7e0f718c13e238e912, name=ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-mon-b, org.label-schema.build-date=20240307, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, RELEASE=wip-yuri5-testing-2024-03-18-1144-e7e80a6, GIT_CLEAN=True, org.label-schema.vendor=CentOS, ceph=True) 2024-03-20T08:41:36.390 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:41:36 smithi194 ceph-mon[27237]: set uid:gid to 167:167 (ceph:ceph) 2024-03-20T08:41:36.390 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:41:36 smithi194 ceph-mon[27237]: ceph version 19.0.0-2153-ge7e80a6a (e7e80a6ac9849f0f23432d0d9832f096730308ae) squid (dev), process ceph-mon, pid 2 2024-03-20T08:41:36.390 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:41:36 smithi194 ceph-mon[27237]: pidfile_write: ignore empty --pid-file 2024-03-20T08:41:36.390 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:41:36 smithi194 ceph-mon[27237]: load: jerasure load: lrc 2024-03-20T08:41:36.390 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:41:36 smithi194 ceph-mon[27237]: rocksdb: RocksDB version: 7.9.2 2024-03-20T08:41:36.390 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:41:36 smithi194 ceph-mon[27237]: rocksdb: Git sha 0 2024-03-20T08:41:36.391 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:41:36 smithi194 ceph-mon[27237]: rocksdb: Compile date 2024-03-18 19:18:37 2024-03-20T08:41:36.391 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:41:36 smithi194 ceph-mon[27237]: rocksdb: DB SUMMARY 2024-03-20T08:41:36.391 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:41:36 smithi194 ceph-mon[27237]: rocksdb: DB Session ID: WJD59CA9O1BT7UZ35J5B 2024-03-20T08:41:36.391 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:41:36 smithi194 ceph-mon[27237]: rocksdb: CURRENT file: CURRENT 2024-03-20T08:41:36.391 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:41:36 smithi194 ceph-mon[27237]: rocksdb: IDENTITY file: IDENTITY 2024-03-20T08:41:36.391 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:41:36 smithi194 ceph-mon[27237]: rocksdb: MANIFEST file: MANIFEST-000005 size: 59 Bytes 2024-03-20T08:41:36.391 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:41:36 smithi194 ceph-mon[27237]: rocksdb: SST files in /var/lib/ceph/mon/ceph-b/store.db dir, Total Num: 0, files: 2024-03-20T08:41:36.391 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:41:36 smithi194 ceph-mon[27237]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-b/store.db: 000004.log size: 511 ; 2024-03-20T08:41:36.391 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:41:36 smithi194 ceph-mon[27237]: rocksdb: Options.error_if_exists: 0 2024-03-20T08:41:36.391 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:41:36 smithi194 ceph-mon[27237]: rocksdb: Options.create_if_missing: 0 2024-03-20T08:41:36.392 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:41:36 smithi194 ceph-mon[27237]: rocksdb: Options.paranoid_checks: 1 2024-03-20T08:41:36.392 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:41:36 smithi194 ceph-mon[27237]: rocksdb: Options.flush_verify_memtable_count: 1 2024-03-20T08:41:36.392 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:41:36 smithi194 ceph-mon[27237]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-03-20T08:41:36.392 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:41:36 smithi194 ceph-mon[27237]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-03-20T08:41:36.392 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:41:36 smithi194 ceph-mon[27237]: rocksdb: Options.env: 0x563e717dfb20 2024-03-20T08:41:36.392 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:41:36 smithi194 ceph-mon[27237]: rocksdb: Options.fs: PosixFileSystem 2024-03-20T08:41:36.392 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:41:36 smithi194 ceph-mon[27237]: rocksdb: Options.info_log: 0x563e73147920 2024-03-20T08:41:36.392 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:41:36 smithi194 ceph-mon[27237]: rocksdb: Options.max_file_opening_threads: 16 2024-03-20T08:41:36.392 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:41:36 smithi194 ceph-mon[27237]: rocksdb: Options.statistics: (nil) 2024-03-20T08:41:36.392 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:41:36 smithi194 ceph-mon[27237]: rocksdb: Options.use_fsync: 0 2024-03-20T08:41:36.393 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:41:36 smithi194 ceph-mon[27237]: rocksdb: Options.max_log_file_size: 0 2024-03-20T08:41:36.393 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:41:36 smithi194 ceph-mon[27237]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-03-20T08:41:36.393 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:41:36 smithi194 ceph-mon[27237]: rocksdb: Options.log_file_time_to_roll: 0 2024-03-20T08:41:36.393 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:41:36 smithi194 ceph-mon[27237]: rocksdb: Options.keep_log_file_num: 1000 2024-03-20T08:41:36.393 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:41:36 smithi194 ceph-mon[27237]: rocksdb: Options.recycle_log_file_num: 0 2024-03-20T08:41:36.393 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:41:36 smithi194 ceph-mon[27237]: rocksdb: Options.allow_fallocate: 1 2024-03-20T08:41:36.393 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:41:36 smithi194 ceph-mon[27237]: rocksdb: Options.allow_mmap_reads: 0 2024-03-20T08:41:36.393 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:41:36 smithi194 ceph-mon[27237]: rocksdb: Options.allow_mmap_writes: 0 2024-03-20T08:41:36.393 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:41:36 smithi194 ceph-mon[27237]: rocksdb: Options.use_direct_reads: 0 2024-03-20T08:41:36.393 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:41:36 smithi194 ceph-mon[27237]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-03-20T08:41:36.393 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:41:36 smithi194 ceph-mon[27237]: rocksdb: Options.create_missing_column_families: 0 2024-03-20T08:41:36.393 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:41:36 smithi194 ceph-mon[27237]: rocksdb: Options.db_log_dir: 2024-03-20T08:41:36.393 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:41:36 smithi194 ceph-mon[27237]: rocksdb: Options.wal_dir: 2024-03-20T08:41:36.393 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:41:36 smithi194 ceph-mon[27237]: rocksdb: Options.table_cache_numshardbits: 6 2024-03-20T08:41:36.394 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:41:36 smithi194 ceph-mon[27237]: rocksdb: Options.WAL_ttl_seconds: 0 2024-03-20T08:41:36.394 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:41:36 smithi194 ceph-mon[27237]: rocksdb: Options.WAL_size_limit_MB: 0 2024-03-20T08:41:36.394 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:41:36 smithi194 ceph-mon[27237]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-03-20T08:41:36.394 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:41:36 smithi194 ceph-mon[27237]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-03-20T08:41:36.394 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:41:36 smithi194 ceph-mon[27237]: rocksdb: Options.is_fd_close_on_exec: 1 2024-03-20T08:41:36.394 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:41:36 smithi194 ceph-mon[27237]: rocksdb: Options.advise_random_on_open: 1 2024-03-20T08:41:36.394 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:41:36 smithi194 ceph-mon[27237]: rocksdb: Options.db_write_buffer_size: 0 2024-03-20T08:41:36.394 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:41:36 smithi194 ceph-mon[27237]: rocksdb: Options.write_buffer_manager: 0x563e7311a1e0 2024-03-20T08:41:36.394 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:41:36 smithi194 ceph-mon[27237]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-03-20T08:41:36.394 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:41:36 smithi194 ceph-mon[27237]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-03-20T08:41:36.394 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:41:36 smithi194 ceph-mon[27237]: rocksdb: Options.use_adaptive_mutex: 0 2024-03-20T08:41:36.394 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:41:36 smithi194 ceph-mon[27237]: rocksdb: Options.rate_limiter: (nil) 2024-03-20T08:41:36.394 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:41:36 smithi194 ceph-mon[27237]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-03-20T08:41:36.394 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:41:36 smithi194 ceph-mon[27237]: rocksdb: Options.wal_recovery_mode: 2 2024-03-20T08:41:36.395 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:41:36 smithi194 ceph-mon[27237]: rocksdb: Options.enable_thread_tracking: 0 2024-03-20T08:41:36.395 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:41:36 smithi194 ceph-mon[27237]: rocksdb: Options.enable_pipelined_write: 0 2024-03-20T08:41:36.395 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:41:36 smithi194 ceph-mon[27237]: rocksdb: Options.unordered_write: 0 2024-03-20T08:41:36.395 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:41:36 smithi194 ceph-mon[27237]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-03-20T08:41:36.395 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:41:36 smithi194 ceph-mon[27237]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-03-20T08:41:36.395 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:41:36 smithi194 ceph-mon[27237]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-03-20T08:41:36.395 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:41:36 smithi194 ceph-mon[27237]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-03-20T08:41:36.395 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:41:36 smithi194 ceph-mon[27237]: rocksdb: Options.row_cache: None 2024-03-20T08:41:36.395 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:41:36 smithi194 ceph-mon[27237]: rocksdb: Options.wal_filter: None 2024-03-20T08:41:36.395 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:41:36 smithi194 ceph-mon[27237]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-03-20T08:41:36.395 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:41:36 smithi194 ceph-mon[27237]: rocksdb: Options.allow_ingest_behind: 0 2024-03-20T08:41:36.395 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:41:36 smithi194 ceph-mon[27237]: rocksdb: Options.two_write_queues: 0 2024-03-20T08:41:36.395 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:41:36 smithi194 ceph-mon[27237]: rocksdb: Options.manual_wal_flush: 0 2024-03-20T08:41:36.395 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:41:36 smithi194 ceph-mon[27237]: rocksdb: Options.wal_compression: 0 2024-03-20T08:41:36.396 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:41:36 smithi194 ceph-mon[27237]: rocksdb: Options.atomic_flush: 0 2024-03-20T08:41:36.396 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:41:36 smithi194 ceph-mon[27237]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-03-20T08:41:36.396 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:41:36 smithi194 ceph-mon[27237]: rocksdb: Options.persist_stats_to_disk: 0 2024-03-20T08:41:36.396 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:41:36 smithi194 ceph-mon[27237]: rocksdb: Options.write_dbid_to_manifest: 0 2024-03-20T08:41:36.396 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:41:36 smithi194 ceph-mon[27237]: rocksdb: Options.log_readahead_size: 0 2024-03-20T08:41:36.396 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:41:36 smithi194 ceph-mon[27237]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-03-20T08:41:36.396 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:41:36 smithi194 ceph-mon[27237]: rocksdb: Options.best_efforts_recovery: 0 2024-03-20T08:41:36.396 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:41:36 smithi194 ceph-mon[27237]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-03-20T08:41:36.396 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:41:36 smithi194 ceph-mon[27237]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-03-20T08:41:36.396 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:41:36 smithi194 ceph-mon[27237]: rocksdb: Options.allow_data_in_errors: 0 2024-03-20T08:41:36.396 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:41:36 smithi194 ceph-mon[27237]: rocksdb: Options.db_host_id: __hostname__ 2024-03-20T08:41:36.396 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:41:36 smithi194 ceph-mon[27237]: rocksdb: Options.enforce_single_del_contracts: true 2024-03-20T08:41:36.396 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:41:36 smithi194 ceph-mon[27237]: rocksdb: Options.max_background_jobs: 2 2024-03-20T08:41:36.397 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:41:36 smithi194 ceph-mon[27237]: rocksdb: Options.max_background_compactions: -1 2024-03-20T08:41:36.397 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:41:36 smithi194 ceph-mon[27237]: rocksdb: Options.max_subcompactions: 1 2024-03-20T08:41:36.397 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:41:36 smithi194 ceph-mon[27237]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-03-20T08:41:36.397 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:41:36 smithi194 ceph-mon[27237]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-03-20T08:41:36.397 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:41:36 smithi194 ceph-mon[27237]: rocksdb: Options.delayed_write_rate : 16777216 2024-03-20T08:41:36.397 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:41:36 smithi194 ceph-mon[27237]: rocksdb: Options.max_total_wal_size: 0 2024-03-20T08:41:36.397 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:41:36 smithi194 ceph-mon[27237]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-03-20T08:41:36.397 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:41:36 smithi194 ceph-mon[27237]: rocksdb: Options.stats_dump_period_sec: 600 2024-03-20T08:41:36.397 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:41:36 smithi194 ceph-mon[27237]: rocksdb: Options.stats_persist_period_sec: 600 2024-03-20T08:41:36.397 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:41:36 smithi194 ceph-mon[27237]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-03-20T08:41:36.397 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:41:36 smithi194 ceph-mon[27237]: rocksdb: Options.max_open_files: -1 2024-03-20T08:41:36.397 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:41:36 smithi194 ceph-mon[27237]: rocksdb: Options.bytes_per_sync: 0 2024-03-20T08:41:36.397 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:41:36 smithi194 ceph-mon[27237]: rocksdb: Options.wal_bytes_per_sync: 0 2024-03-20T08:41:36.397 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:41:36 smithi194 ceph-mon[27237]: rocksdb: Options.strict_bytes_per_sync: 0 2024-03-20T08:41:36.398 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:41:36 smithi194 ceph-mon[27237]: rocksdb: Options.compaction_readahead_size: 0 2024-03-20T08:41:36.398 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:41:36 smithi194 ceph-mon[27237]: rocksdb: Options.max_background_flushes: -1 2024-03-20T08:41:36.398 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:41:36 smithi194 ceph-mon[27237]: rocksdb: Compression algorithms supported: 2024-03-20T08:41:36.398 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:41:36 smithi194 ceph-mon[27237]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-03-20T08:41:36.398 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:41:36 smithi194 ceph-mon[27237]: rocksdb: kZSTD supported: 0 2024-03-20T08:41:36.398 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:41:36 smithi194 ceph-mon[27237]: rocksdb: kXpressCompression supported: 0 2024-03-20T08:41:36.398 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:41:36 smithi194 ceph-mon[27237]: rocksdb: kLZ4HCCompression supported: 1 2024-03-20T08:41:36.398 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:41:36 smithi194 ceph-mon[27237]: rocksdb: kZlibCompression supported: 1 2024-03-20T08:41:36.398 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:41:36 smithi194 ceph-mon[27237]: rocksdb: kSnappyCompression supported: 1 2024-03-20T08:41:36.398 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:41:36 smithi194 ceph-mon[27237]: rocksdb: kLZ4Compression supported: 1 2024-03-20T08:41:36.398 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:41:36 smithi194 ceph-mon[27237]: rocksdb: kBZip2Compression supported: 0 2024-03-20T08:41:36.398 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:41:36 smithi194 ceph-mon[27237]: rocksdb: Fast CRC32 supported: Supported on x86 2024-03-20T08:41:36.398 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:41:36 smithi194 ceph-mon[27237]: rocksdb: DMutex implementation: pthread_mutex_t 2024-03-20T08:41:36.398 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:41:36 smithi194 ceph-mon[27237]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-b/store.db/MANIFEST-000005 2024-03-20T08:41:36.399 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:41:36 smithi194 ceph-mon[27237]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-03-20T08:41:36.399 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:41:36 smithi194 ceph-mon[27237]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-03-20T08:41:36.399 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:41:36 smithi194 ceph-mon[27237]: rocksdb: Options.merge_operator: 2024-03-20T08:41:36.399 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:41:36 smithi194 ceph-mon[27237]: rocksdb: Options.compaction_filter: None 2024-03-20T08:41:36.399 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:41:36 smithi194 ceph-mon[27237]: rocksdb: Options.compaction_filter_factory: None 2024-03-20T08:41:36.399 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:41:36 smithi194 ceph-mon[27237]: rocksdb: Options.sst_partitioner_factory: None 2024-03-20T08:41:36.399 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:41:36 smithi194 ceph-mon[27237]: rocksdb: Options.memtable_factory: SkipListFactory 2024-03-20T08:41:36.399 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:41:36 smithi194 ceph-mon[27237]: rocksdb: Options.table_factory: BlockBasedTable 2024-03-20T08:41:36.399 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:41:36 smithi194 ceph-mon[27237]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x563e73147a20) 2024-03-20T08:41:36.399 INFO:journalctl@ceph.mon.b.smithi194.stdout: cache_index_and_filter_blocks: 1 2024-03-20T08:41:36.399 INFO:journalctl@ceph.mon.b.smithi194.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-03-20T08:41:36.399 INFO:journalctl@ceph.mon.b.smithi194.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-03-20T08:41:36.399 INFO:journalctl@ceph.mon.b.smithi194.stdout: pin_top_level_index_and_filter: 1 2024-03-20T08:41:36.399 INFO:journalctl@ceph.mon.b.smithi194.stdout: index_type: 0 2024-03-20T08:41:36.399 INFO:journalctl@ceph.mon.b.smithi194.stdout: data_block_index_type: 0 2024-03-20T08:41:36.400 INFO:journalctl@ceph.mon.b.smithi194.stdout: index_shortening: 1 2024-03-20T08:41:36.400 INFO:journalctl@ceph.mon.b.smithi194.stdout: data_block_hash_table_util_ratio: 0.750000 2024-03-20T08:41:36.400 INFO:journalctl@ceph.mon.b.smithi194.stdout: checksum: 4 2024-03-20T08:41:36.400 INFO:journalctl@ceph.mon.b.smithi194.stdout: no_block_cache: 0 2024-03-20T08:41:36.400 INFO:journalctl@ceph.mon.b.smithi194.stdout: block_cache: 0x563e7314d090 2024-03-20T08:41:36.400 INFO:journalctl@ceph.mon.b.smithi194.stdout: block_cache_name: BinnedLRUCache 2024-03-20T08:41:36.400 INFO:journalctl@ceph.mon.b.smithi194.stdout: block_cache_options: 2024-03-20T08:41:36.400 INFO:journalctl@ceph.mon.b.smithi194.stdout: capacity : 536870912 2024-03-20T08:41:36.400 INFO:journalctl@ceph.mon.b.smithi194.stdout: num_shard_bits : 4 2024-03-20T08:41:36.400 INFO:journalctl@ceph.mon.b.smithi194.stdout: strict_capacity_limit : 0 2024-03-20T08:41:36.400 INFO:journalctl@ceph.mon.b.smithi194.stdout: high_pri_pool_ratio: 0.000 2024-03-20T08:41:36.400 INFO:journalctl@ceph.mon.b.smithi194.stdout: block_cache_compressed: (nil) 2024-03-20T08:41:36.400 INFO:journalctl@ceph.mon.b.smithi194.stdout: persistent_cache: (nil) 2024-03-20T08:41:36.400 INFO:journalctl@ceph.mon.b.smithi194.stdout: block_size: 4096 2024-03-20T08:41:36.400 INFO:journalctl@ceph.mon.b.smithi194.stdout: block_size_deviation: 10 2024-03-20T08:41:36.401 INFO:journalctl@ceph.mon.b.smithi194.stdout: block_restart_interval: 16 2024-03-20T08:41:36.401 INFO:journalctl@ceph.mon.b.smithi194.stdout: index_block_restart_interval: 1 2024-03-20T08:41:36.401 INFO:journalctl@ceph.mon.b.smithi194.stdout: metadata_block_size: 4096 2024-03-20T08:41:36.401 INFO:journalctl@ceph.mon.b.smithi194.stdout: partition_filters: 0 2024-03-20T08:41:36.401 INFO:journalctl@ceph.mon.b.smithi194.stdout: use_delta_encoding: 1 2024-03-20T08:41:36.401 INFO:journalctl@ceph.mon.b.smithi194.stdout: filter_policy: bloomfilter 2024-03-20T08:41:36.401 INFO:journalctl@ceph.mon.b.smithi194.stdout: whole_key_filtering: 1 2024-03-20T08:41:36.401 INFO:journalctl@ceph.mon.b.smithi194.stdout: verify_compression: 0 2024-03-20T08:41:36.401 INFO:journalctl@ceph.mon.b.smithi194.stdout: read_amp_bytes_per_bit: 0 2024-03-20T08:41:36.401 INFO:journalctl@ceph.mon.b.smithi194.stdout: format_version: 5 2024-03-20T08:41:36.401 INFO:journalctl@ceph.mon.b.smithi194.stdout: enable_index_compression: 1 2024-03-20T08:41:36.401 INFO:journalctl@ceph.mon.b.smithi194.stdout: block_align: 0 2024-03-20T08:41:36.401 INFO:journalctl@ceph.mon.b.smithi194.stdout: max_auto_readahead_size: 262144 2024-03-20T08:41:36.401 INFO:journalctl@ceph.mon.b.smithi194.stdout: prepopulate_block_cache: 0 2024-03-20T08:41:36.401 INFO:journalctl@ceph.mon.b.smithi194.stdout: initial_auto_readahead_size: 8192 2024-03-20T08:41:36.408 INFO:journalctl@ceph.mon.b.smithi194.stdout: num_file_reads_for_auto_readahead: 2 2024-03-20T08:41:36.408 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:41:36 smithi194 ceph-mon[27237]: rocksdb: Options.write_buffer_size: 33554432 2024-03-20T08:41:36.408 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:41:36 smithi194 ceph-mon[27237]: rocksdb: Options.max_write_buffer_number: 2 2024-03-20T08:41:36.408 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:41:36 smithi194 ceph-mon[27237]: rocksdb: Options.compression: NoCompression 2024-03-20T08:41:36.408 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:41:36 smithi194 ceph-mon[27237]: rocksdb: Options.bottommost_compression: Disabled 2024-03-20T08:41:36.408 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:41:36 smithi194 ceph-mon[27237]: rocksdb: Options.prefix_extractor: nullptr 2024-03-20T08:41:36.409 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:41:36 smithi194 ceph-mon[27237]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-03-20T08:41:36.409 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:41:36 smithi194 ceph-mon[27237]: rocksdb: Options.num_levels: 7 2024-03-20T08:41:36.409 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:41:36 smithi194 ceph-mon[27237]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-03-20T08:41:36.409 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:41:36 smithi194 ceph-mon[27237]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-03-20T08:41:36.409 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:41:36 smithi194 ceph-mon[27237]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-03-20T08:41:36.409 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:41:36 smithi194 ceph-mon[27237]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-03-20T08:41:36.409 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:41:36 smithi194 ceph-mon[27237]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-03-20T08:41:36.409 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:41:36 smithi194 ceph-mon[27237]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-03-20T08:41:36.409 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:41:36 smithi194 ceph-mon[27237]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-03-20T08:41:36.409 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:41:36 smithi194 ceph-mon[27237]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-03-20T08:41:36.409 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:41:36 smithi194 ceph-mon[27237]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-03-20T08:41:36.409 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:41:36 smithi194 ceph-mon[27237]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-03-20T08:41:36.409 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:41:36 smithi194 ceph-mon[27237]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-03-20T08:41:36.410 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:41:36 smithi194 ceph-mon[27237]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-03-20T08:41:36.410 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:41:36 smithi194 ceph-mon[27237]: rocksdb: Options.compression_opts.window_bits: -14 2024-03-20T08:41:36.410 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:41:36 smithi194 ceph-mon[27237]: rocksdb: Options.compression_opts.level: 32767 2024-03-20T08:41:36.410 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:41:36 smithi194 ceph-mon[27237]: rocksdb: Options.compression_opts.strategy: 0 2024-03-20T08:41:36.410 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:41:36 smithi194 ceph-mon[27237]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-03-20T08:41:36.410 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:41:36 smithi194 ceph-mon[27237]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-03-20T08:41:36.410 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:41:36 smithi194 ceph-mon[27237]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-03-20T08:41:36.410 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:41:36 smithi194 ceph-mon[27237]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-03-20T08:41:36.410 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:41:36 smithi194 ceph-mon[27237]: rocksdb: Options.compression_opts.enabled: false 2024-03-20T08:41:36.410 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:41:36 smithi194 ceph-mon[27237]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-03-20T08:41:36.411 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:41:36 smithi194 ceph-mon[27237]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-03-20T08:41:36.411 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:41:36 smithi194 ceph-mon[27237]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-03-20T08:41:36.411 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:41:36 smithi194 ceph-mon[27237]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-03-20T08:41:36.411 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:41:36 smithi194 ceph-mon[27237]: rocksdb: Options.target_file_size_base: 67108864 2024-03-20T08:41:36.411 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:41:36 smithi194 ceph-mon[27237]: rocksdb: Options.target_file_size_multiplier: 1 2024-03-20T08:41:36.411 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:41:36 smithi194 ceph-mon[27237]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-03-20T08:41:36.411 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:41:36 smithi194 ceph-mon[27237]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-03-20T08:41:36.411 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:41:36 smithi194 ceph-mon[27237]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-03-20T08:41:36.411 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:41:36 smithi194 ceph-mon[27237]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-03-20T08:41:36.411 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:41:36 smithi194 ceph-mon[27237]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-03-20T08:41:36.411 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:41:36 smithi194 ceph-mon[27237]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-03-20T08:41:36.411 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:41:36 smithi194 ceph-mon[27237]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-03-20T08:41:36.411 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:41:36 smithi194 ceph-mon[27237]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-03-20T08:41:36.412 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:41:36 smithi194 ceph-mon[27237]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-03-20T08:41:36.412 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:41:36 smithi194 ceph-mon[27237]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-03-20T08:41:36.412 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:41:36 smithi194 ceph-mon[27237]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-03-20T08:41:36.412 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:41:36 smithi194 ceph-mon[27237]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-03-20T08:41:36.412 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:41:36 smithi194 ceph-mon[27237]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-03-20T08:41:36.412 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:41:36 smithi194 ceph-mon[27237]: rocksdb: Options.arena_block_size: 1048576 2024-03-20T08:41:36.412 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:41:36 smithi194 ceph-mon[27237]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-03-20T08:41:36.412 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:41:36 smithi194 ceph-mon[27237]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-03-20T08:41:36.412 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:41:36 smithi194 ceph-mon[27237]: rocksdb: Options.disable_auto_compactions: 0 2024-03-20T08:41:36.412 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:41:36 smithi194 ceph-mon[27237]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-03-20T08:41:36.412 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:41:36 smithi194 ceph-mon[27237]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-03-20T08:41:36.412 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:41:36 smithi194 ceph-mon[27237]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-03-20T08:41:36.412 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:41:36 smithi194 ceph-mon[27237]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-03-20T08:41:36.412 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:41:36 smithi194 ceph-mon[27237]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-03-20T08:41:36.412 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:41:36 smithi194 ceph-mon[27237]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-03-20T08:41:36.413 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:41:36 smithi194 ceph-mon[27237]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-03-20T08:41:36.413 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:41:36 smithi194 ceph-mon[27237]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-03-20T08:41:36.413 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:41:36 smithi194 ceph-mon[27237]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-03-20T08:41:36.413 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:41:36 smithi194 ceph-mon[27237]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-03-20T08:41:36.413 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:41:36 smithi194 ceph-mon[27237]: rocksdb: Options.table_properties_collectors: 2024-03-20T08:41:36.413 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:41:36 smithi194 ceph-mon[27237]: rocksdb: Options.inplace_update_support: 0 2024-03-20T08:41:36.413 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:41:36 smithi194 ceph-mon[27237]: rocksdb: Options.inplace_update_num_locks: 10000 2024-03-20T08:41:36.413 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:41:36 smithi194 ceph-mon[27237]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-03-20T08:41:36.413 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:41:36 smithi194 ceph-mon[27237]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-03-20T08:41:36.413 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:41:36 smithi194 ceph-mon[27237]: rocksdb: Options.memtable_huge_page_size: 0 2024-03-20T08:41:36.413 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:41:36 smithi194 ceph-mon[27237]: rocksdb: Options.bloom_locality: 0 2024-03-20T08:41:36.413 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:41:36 smithi194 ceph-mon[27237]: rocksdb: Options.max_successive_merges: 0 2024-03-20T08:41:36.413 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:41:36 smithi194 ceph-mon[27237]: rocksdb: Options.optimize_filters_for_hits: 0 2024-03-20T08:41:36.413 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:41:36 smithi194 ceph-mon[27237]: rocksdb: Options.paranoid_file_checks: 0 2024-03-20T08:41:36.414 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:41:36 smithi194 ceph-mon[27237]: rocksdb: Options.force_consistency_checks: 1 2024-03-20T08:41:36.414 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:41:36 smithi194 ceph-mon[27237]: rocksdb: Options.report_bg_io_stats: 0 2024-03-20T08:41:36.414 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:41:36 smithi194 ceph-mon[27237]: rocksdb: Options.ttl: 2592000 2024-03-20T08:41:36.414 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:41:36 smithi194 ceph-mon[27237]: rocksdb: Options.periodic_compaction_seconds: 0 2024-03-20T08:41:36.414 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:41:36 smithi194 ceph-mon[27237]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-03-20T08:41:36.414 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:41:36 smithi194 ceph-mon[27237]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-03-20T08:41:36.414 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:41:36 smithi194 ceph-mon[27237]: rocksdb: Options.enable_blob_files: false 2024-03-20T08:41:36.414 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:41:36 smithi194 ceph-mon[27237]: rocksdb: Options.min_blob_size: 0 2024-03-20T08:41:36.414 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:41:36 smithi194 ceph-mon[27237]: rocksdb: Options.blob_file_size: 268435456 2024-03-20T08:41:36.414 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:41:36 smithi194 ceph-mon[27237]: rocksdb: Options.blob_compression_type: NoCompression 2024-03-20T08:41:36.414 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:41:36 smithi194 ceph-mon[27237]: rocksdb: Options.enable_blob_garbage_collection: false 2024-03-20T08:41:36.414 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:41:36 smithi194 ceph-mon[27237]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-03-20T08:41:36.414 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:41:36 smithi194 ceph-mon[27237]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-03-20T08:41:36.414 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:41:36 smithi194 ceph-mon[27237]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-03-20T08:41:36.415 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:41:36 smithi194 ceph-mon[27237]: rocksdb: Options.blob_file_starting_level: 0 2024-03-20T08:41:36.415 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:41:36 smithi194 ceph-mon[27237]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-03-20T08:41:36.415 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:41:36 smithi194 ceph-mon[27237]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-b/store.db/MANIFEST-000005 succeeded,manifest_file_number is 5, next_file_number is 7, last_sequence is 0, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-03-20T08:41:36.415 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:41:36 smithi194 ceph-mon[27237]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 0 2024-03-20T08:41:36.415 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:41:36 smithi194 ceph-mon[27237]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: 4c5d4e3b-7d75-495c-86c6-6fa83d3e4aa1 2024-03-20T08:41:36.415 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:41:36 smithi194 ceph-mon[27237]: rocksdb: EVENT_LOG_v1 {"time_micros": 1710924096160138, "job": 1, "event": "recovery_started", "wal_files": [4]} 2024-03-20T08:41:36.415 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:41:36 smithi194 ceph-mon[27237]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #4 mode 2 2024-03-20T08:41:36.415 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:41:36 smithi194 ceph-mon[27237]: rocksdb: EVENT_LOG_v1 {"time_micros": 1710924096160552, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 8, "file_size": 1622, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 1, "largest_seqno": 5, "table_properties": {"data_size": 523, "index_size": 31, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 69, "raw_key_size": 115, "raw_average_key_size": 23, "raw_value_size": 401, "raw_average_value_size": 80, "num_data_blocks": 1, "num_entries": 5, "num_filter_entries": 5, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1710924096, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "4c5d4e3b-7d75-495c-86c6-6fa83d3e4aa1", "db_session_id": "WJD59CA9O1BT7UZ35J5B", "orig_file_number": 8, "seqno_to_time_mapping": "N/A"}} 2024-03-20T08:41:36.415 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:41:36 smithi194 ceph-mon[27237]: rocksdb: EVENT_LOG_v1 {"time_micros": 1710924096160631, "job": 1, "event": "recovery_finished"} 2024-03-20T08:41:36.415 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:41:36 smithi194 ceph-mon[27237]: rocksdb: [db/version_set.cc:5047] Creating manifest 10 2024-03-20T08:41:36.415 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:41:36 smithi194 ceph-mon[27237]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-b/store.db/000004.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-03-20T08:41:36.415 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:41:36 smithi194 ceph-mon[27237]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x563e73240000 2024-03-20T08:41:36.416 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:41:36 smithi194 ceph-mon[27237]: rocksdb: DB pointer 0x563e7322a000 2024-03-20T08:41:36.416 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:41:36 smithi194 ceph-mon[27237]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-03-20T08:41:36.416 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:41:36 smithi194 ceph-mon[27237]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-03-20T08:41:36.416 INFO:journalctl@ceph.mon.b.smithi194.stdout: ** DB Stats ** 2024-03-20T08:41:36.416 INFO:journalctl@ceph.mon.b.smithi194.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-03-20T08:41:36.416 INFO:journalctl@ceph.mon.b.smithi194.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-03-20T08:41:36.416 INFO:journalctl@ceph.mon.b.smithi194.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-03-20T08:41:36.416 INFO:journalctl@ceph.mon.b.smithi194.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-03-20T08:41:36.416 INFO:journalctl@ceph.mon.b.smithi194.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-03-20T08:41:36.416 INFO:journalctl@ceph.mon.b.smithi194.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-03-20T08:41:36.416 INFO:journalctl@ceph.mon.b.smithi194.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-03-20T08:41:36.416 INFO:journalctl@ceph.mon.b.smithi194.stdout: 2024-03-20T08:41:36.416 INFO:journalctl@ceph.mon.b.smithi194.stdout: ** Compaction Stats [default] ** 2024-03-20T08:41:36.416 INFO:journalctl@ceph.mon.b.smithi194.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-03-20T08:41:36.416 INFO:journalctl@ceph.mon.b.smithi194.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-03-20T08:41:36.417 INFO:journalctl@ceph.mon.b.smithi194.stdout: L0 1/0 1.58 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 3.9 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-03-20T08:41:36.417 INFO:journalctl@ceph.mon.b.smithi194.stdout: Sum 1/0 1.58 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 3.9 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-03-20T08:41:36.417 INFO:journalctl@ceph.mon.b.smithi194.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 3.9 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-03-20T08:41:36.417 INFO:journalctl@ceph.mon.b.smithi194.stdout: 2024-03-20T08:41:36.417 INFO:journalctl@ceph.mon.b.smithi194.stdout: ** Compaction Stats [default] ** 2024-03-20T08:41:36.417 INFO:journalctl@ceph.mon.b.smithi194.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-03-20T08:41:36.417 INFO:journalctl@ceph.mon.b.smithi194.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-03-20T08:41:36.417 INFO:journalctl@ceph.mon.b.smithi194.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 3.9 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-03-20T08:41:36.417 INFO:journalctl@ceph.mon.b.smithi194.stdout: 2024-03-20T08:41:36.417 INFO:journalctl@ceph.mon.b.smithi194.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-03-20T08:41:36.417 INFO:journalctl@ceph.mon.b.smithi194.stdout: 2024-03-20T08:41:36.417 INFO:journalctl@ceph.mon.b.smithi194.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-03-20T08:41:36.418 INFO:journalctl@ceph.mon.b.smithi194.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-03-20T08:41:36.418 INFO:journalctl@ceph.mon.b.smithi194.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-03-20T08:41:36.418 INFO:journalctl@ceph.mon.b.smithi194.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-03-20T08:41:36.418 INFO:journalctl@ceph.mon.b.smithi194.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-03-20T08:41:36.418 INFO:journalctl@ceph.mon.b.smithi194.stdout: AddFile(Keys): cumulative 0, interval 0 2024-03-20T08:41:36.418 INFO:journalctl@ceph.mon.b.smithi194.stdout: Cumulative compaction: 0.00 GB write, 0.63 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-20T08:41:36.418 INFO:journalctl@ceph.mon.b.smithi194.stdout: Interval compaction: 0.00 GB write, 0.63 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-20T08:41:36.418 INFO:journalctl@ceph.mon.b.smithi194.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-03-20T08:41:36.418 INFO:journalctl@ceph.mon.b.smithi194.stdout: Block cache BinnedLRUCache@0x563e7314d090#2 capacity: 512.00 MB usage: 0.22 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 9e-06 secs_since: 0 2024-03-20T08:41:36.418 INFO:journalctl@ceph.mon.b.smithi194.stdout: Block cache entry stats(count,size,portion): FilterBlock(1,0.11 KB,2.08616e-05%) IndexBlock(1,0.11 KB,2.08616e-05%) Misc(1,0.00 KB,0%) 2024-03-20T08:41:36.418 INFO:journalctl@ceph.mon.b.smithi194.stdout: 2024-03-20T08:41:36.418 INFO:journalctl@ceph.mon.b.smithi194.stdout: ** File Read Latency Histogram By Level [default] ** 2024-03-20T08:41:36.418 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:41:36 smithi194 ceph-mon[27237]: mon.b does not exist in monmap, will attempt to join an existing cluster 2024-03-20T08:41:36.419 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:41:36 smithi194 ceph-mon[27237]: using public_addr v2:172.21.15.194:0/0 -> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] 2024-03-20T08:41:36.419 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:41:36 smithi194 ceph-mon[27237]: starting mon.b rank -1 at public addrs [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] at bind addrs [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] mon_data /var/lib/ceph/mon/ceph-b fsid 4c3f3eba-e695-11ee-95c9-87774f69a715 2024-03-20T08:41:36.419 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:41:36 smithi194 ceph-mon[27237]: mon.b@-1(???) e0 preinit fsid 4c3f3eba-e695-11ee-95c9-87774f69a715 2024-03-20T08:41:36.419 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:41:36 smithi194 ceph-mon[27237]: mon.b@-1(synchronizing).mds e1 new map 2024-03-20T08:41:36.419 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:41:36 smithi194 ceph-mon[27237]: mon.b@-1(synchronizing).mds e1 print_map 2024-03-20T08:41:36.419 INFO:journalctl@ceph.mon.b.smithi194.stdout: e1 2024-03-20T08:41:36.419 INFO:journalctl@ceph.mon.b.smithi194.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-03-20T08:41:36.419 INFO:journalctl@ceph.mon.b.smithi194.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments} 2024-03-20T08:41:36.419 INFO:journalctl@ceph.mon.b.smithi194.stdout: legacy client fscid: -1 2024-03-20T08:41:36.419 INFO:journalctl@ceph.mon.b.smithi194.stdout: 2024-03-20T08:41:36.419 INFO:journalctl@ceph.mon.b.smithi194.stdout: No filesystems configured 2024-03-20T08:41:36.419 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:41:36 smithi194 ceph-mon[27237]: mon.b@-1(synchronizing).osd e0 _set_cache_ratios kv ratio 0.25 inc ratio 0.375 full ratio 0.375 2024-03-20T08:41:36.419 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:41:36 smithi194 ceph-mon[27237]: mon.b@-1(synchronizing).osd e0 register_cache_with_pcm pcm target: 2147483648 pcm max: 1020054732 pcm min: 134217728 inc_osd_cache size: 1 2024-03-20T08:41:36.420 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:41:36 smithi194 ceph-mon[27237]: mon.b@-1(synchronizing).osd e1 e1: 0 total, 0 up, 0 in 2024-03-20T08:41:36.420 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:41:36 smithi194 ceph-mon[27237]: mon.b@-1(synchronizing).osd e2 e2: 0 total, 0 up, 0 in 2024-03-20T08:41:36.420 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:41:36 smithi194 ceph-mon[27237]: mon.b@-1(synchronizing).osd e3 e3: 0 total, 0 up, 0 in 2024-03-20T08:41:36.420 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:41:36 smithi194 ceph-mon[27237]: mon.b@-1(synchronizing).osd e4 e4: 0 total, 0 up, 0 in 2024-03-20T08:41:36.420 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:41:36 smithi194 ceph-mon[27237]: mon.b@-1(synchronizing).osd e4 crush map has features 3314932999778484224, adjusting msgr requires 2024-03-20T08:41:36.420 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:41:36 smithi194 ceph-mon[27237]: mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-03-20T08:41:36.420 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:41:36 smithi194 ceph-mon[27237]: mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-03-20T08:41:36.420 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:41:36 smithi194 ceph-mon[27237]: mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-03-20T08:41:36.420 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:41:36 smithi194 ceph-mon[27237]: Deploying cephadm binary to smithi194 2024-03-20T08:41:36.420 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:41:36 smithi194 ceph-mon[27237]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-20T08:41:36.420 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:41:36 smithi194 ceph-mon[27237]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-20T08:41:36.420 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:41:36 smithi194 ceph-mon[27237]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:41:36.421 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:41:36 smithi194 ceph-mon[27237]: Added host smithi194 2024-03-20T08:41:36.421 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:41:36 smithi194 ceph-mon[27237]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-20T08:41:36.421 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:41:36 smithi194 ceph-mon[27237]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:41:36.421 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:41:36 smithi194 ceph-mon[27237]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-20T08:41:36.421 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:41:36 smithi194 ceph-mon[27237]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:41:36.421 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:41:36 smithi194 ceph-mon[27237]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:41:36.421 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:41:36 smithi194 ceph-mon[27237]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi194", "name": "osd_memory_target"}]: dispatch 2024-03-20T08:41:36.421 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:41:36 smithi194 ceph-mon[27237]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-20T08:41:36.421 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:41:36 smithi194 ceph-mon[27237]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-20T08:41:36.421 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:41:36 smithi194 ceph-mon[27237]: Updating smithi194:/etc/ceph/ceph.conf 2024-03-20T08:41:36.421 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:41:36 smithi194 ceph-mon[27237]: Updating smithi194:/var/lib/ceph/4c3f3eba-e695-11ee-95c9-87774f69a715/config/ceph.conf 2024-03-20T08:41:36.421 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:41:36 smithi194 ceph-mon[27237]: Updating smithi194:/etc/ceph/ceph.client.admin.keyring 2024-03-20T08:41:36.422 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:41:36 smithi194 ceph-mon[27237]: Updating smithi194:/var/lib/ceph/4c3f3eba-e695-11ee-95c9-87774f69a715/config/ceph.client.admin.keyring 2024-03-20T08:41:36.422 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:41:36 smithi194 ceph-mon[27237]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:41:36.422 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:41:36 smithi194 ceph-mon[27237]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:41:36.422 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:41:36 smithi194 ceph-mon[27237]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.agent.smithi194", "caps": []}]: dispatch 2024-03-20T08:41:36.422 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:41:36 smithi194 ceph-mon[27237]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.agent.smithi194", "caps": []}]': finished 2024-03-20T08:41:36.422 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:41:36 smithi194 ceph-mon[27237]: Deploying daemon agent.smithi194 on smithi194 2024-03-20T08:41:36.422 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:41:36 smithi194 ceph-mon[27237]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-20T08:41:36.422 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:41:36 smithi194 ceph-mon[27237]: from='client.14180 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-20T08:41:36.422 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:41:36 smithi194 ceph-mon[27237]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:41:36.422 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:41:36 smithi194 ceph-mon[27237]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:41:36.422 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:41:36 smithi194 ceph-mon[27237]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:41:36.422 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:41:36 smithi194 ceph-mon[27237]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-20T08:41:36.422 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:41:36 smithi194 ceph-mon[27237]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-20T08:41:36.423 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:41:36 smithi194 ceph-mon[27237]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-20T08:41:36.423 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:41:36 smithi194 ceph-mon[27237]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:41:36.423 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:41:36 smithi194 ceph-mon[27237]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-20T08:41:36.423 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:41:36 smithi194 ceph-mon[27237]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-20T08:41:36.423 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:41:36 smithi194 ceph-mon[27237]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-20T08:41:36.423 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:41:36 smithi194 ceph-mon[27237]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:41:36.423 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:41:36 smithi194 ceph-mon[27237]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:41:36.423 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:41:36 smithi194 ceph-mon[27237]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:41:36.423 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:41:36 smithi194 ceph-mon[27237]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:41:36.423 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:41:36 smithi194 ceph-mon[27237]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-20T08:41:36.423 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:41:36 smithi194 ceph-mon[27237]: from='client.? 172.21.15.183:0/21530245' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-03-20T08:41:36.423 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:41:36 smithi194 ceph-mon[27237]: from='client.? 172.21.15.183:0/21530245' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-03-20T08:41:36.424 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:41:36 smithi194 ceph-mon[27237]: osdmap e4: 0 total, 0 up, 0 in 2024-03-20T08:41:36.424 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:41:36 smithi194 ceph-mon[27237]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:41:36.424 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:41:36 smithi194 ceph-mon[27237]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:41:36.424 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:41:36 smithi194 ceph-mon[27237]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-20T08:41:36.424 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:41:36 smithi194 ceph-mon[27237]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:41:36.424 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:41:36 smithi194 ceph-mon[27237]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:41:36.424 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:41:36 smithi194 ceph-mon[27237]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:41:36.424 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:41:36 smithi194 ceph-mon[27237]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:41:36.424 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:41:36 smithi194 ceph-mon[27237]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-20T08:41:36.424 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:41:36 smithi194 ceph-mon[27237]: from='client.14184 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi183:172.21.15.183=a;smithi183:[v2:172.21.15.183:3301,v1:172.21.15.183:6790]=c;smithi194:172.21.15.194=b", "target": ["mon-mgr", ""]}]: dispatch 2024-03-20T08:41:36.424 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:41:36 smithi194 ceph-mon[27237]: Saving service mon spec with placement smithi183:172.21.15.183=a;smithi183:[v2:172.21.15.183:3301,v1:172.21.15.183:6790]=c;smithi194:172.21.15.194=b;count:3 2024-03-20T08:41:36.424 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:41:36 smithi194 ceph-mon[27237]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:41:36.424 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:41:36 smithi194 ceph-mon[27237]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-20T08:41:36.425 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:41:36 smithi194 ceph-mon[27237]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-20T08:41:36.425 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:41:36 smithi194 ceph-mon[27237]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-20T08:41:36.425 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:41:36 smithi194 ceph-mon[27237]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:41:36.425 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:41:36 smithi194 ceph-mon[27237]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-20T08:41:36.425 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:41:36 smithi194 ceph-mon[27237]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-20T08:41:36.425 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:41:36 smithi194 ceph-mon[27237]: Deploying daemon mon.b on smithi194 2024-03-20T08:41:36.425 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:41:36 smithi194 ceph-mon[27237]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-20T08:41:36.425 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:41:36 smithi194 ceph-mon[27237]: mon.b@-1(synchronizing).paxosservice(auth 1..5) refresh upgraded, format 0 -> 3 2024-03-20T08:41:36.425 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:41:36 smithi194 bash[27189]: da6a20d027a94b6237e2f032b342d32e508375805cb7e427ff891e7d6919fe6a 2024-03-20T08:41:36.425 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:41:36 smithi194 systemd[1]: Started Ceph mon.b for 4c3f3eba-e695-11ee-95c9-87774f69a715. 2024-03-20T08:41:41.233 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-03-20T08:41:41.233 INFO:teuthology.orchestra.run.smithi194.stdout:{"epoch":2,"fsid":"4c3f3eba-e695-11ee-95c9-87774f69a715","modified":"2024-03-20T08:41:36.184066Z","created":"2024-03-20T08:39:56.427572Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:3300","nonce":0},{"type":"v1","addr":"172.21.15.183:6789","nonce":0}]},"addr":"172.21.15.183:6789/0","public_addr":"172.21.15.183:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:3300","nonce":0},{"type":"v1","addr":"172.21.15.194:6789","nonce":0}]},"addr":"172.21.15.194:6789/0","public_addr":"172.21.15.194:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2024-03-20T08:41:41.235 INFO:teuthology.orchestra.run.smithi194.stderr:dumped monmap epoch 2 2024-03-20T08:41:41.545 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:41:41 smithi194 ceph-mon[27237]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-20T08:41:41.546 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:41:41 smithi194 ceph-mon[27237]: mon.a calling monitor election 2024-03-20T08:41:41.546 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:41:41 smithi194 ceph-mon[27237]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-20T08:41:41.546 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:41:41 smithi194 ceph-mon[27237]: from='client.? 172.21.15.194:0/1341410218' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-20T08:41:41.546 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:41:41 smithi194 ceph-mon[27237]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-20T08:41:41.546 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:41:41 smithi194 ceph-mon[27237]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-20T08:41:41.546 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:41:41 smithi194 ceph-mon[27237]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-20T08:41:41.546 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:41:41 smithi194 ceph-mon[27237]: mon.b calling monitor election 2024-03-20T08:41:41.546 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:41:41 smithi194 ceph-mon[27237]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-20T08:41:41.546 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:41:41 smithi194 ceph-mon[27237]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-20T08:41:41.546 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:41:41 smithi194 ceph-mon[27237]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-20T08:41:41.546 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:41:41 smithi194 ceph-mon[27237]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-20T08:41:41.547 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:41:41 smithi194 ceph-mon[27237]: mon.a is new leader, mons a,b in quorum (ranks 0,1) 2024-03-20T08:41:41.547 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:41:41 smithi194 ceph-mon[27237]: monmap e2: 2 mons at {a=[v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0],b=[v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-20T08:41:41.547 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:41:41 smithi194 ceph-mon[27237]: fsmap 2024-03-20T08:41:41.547 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:41:41 smithi194 ceph-mon[27237]: osdmap e4: 0 total, 0 up, 0 in 2024-03-20T08:41:41.547 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:41:41 smithi194 ceph-mon[27237]: mgrmap e13: y(active, since 41s) 2024-03-20T08:41:41.547 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:41:41 smithi194 ceph-mon[27237]: overall HEALTH_OK 2024-03-20T08:41:41.547 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:41:41 smithi194 ceph-mon[27237]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:41:41.547 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:41:41 smithi194 ceph-mon[27237]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:41:41.547 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:41:41 smithi194 ceph-mon[27237]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:41:41.547 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:41:41 smithi194 ceph-mon[27237]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-20T08:41:41.547 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:41:41 smithi194 ceph-mon[27237]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-20T08:41:41.704 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:41:41 smithi183 ceph-mon[25285]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-20T08:41:41.704 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:41:41 smithi183 ceph-mon[25285]: mon.a calling monitor election 2024-03-20T08:41:41.704 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:41:41 smithi183 ceph-mon[25285]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-20T08:41:41.705 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:41:41 smithi183 ceph-mon[25285]: from='client.? 172.21.15.194:0/1341410218' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-20T08:41:41.705 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:41:41 smithi183 ceph-mon[25285]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-20T08:41:41.705 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:41:41 smithi183 ceph-mon[25285]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-20T08:41:41.705 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:41:41 smithi183 ceph-mon[25285]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-20T08:41:41.705 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:41:41 smithi183 ceph-mon[25285]: mon.b calling monitor election 2024-03-20T08:41:41.705 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:41:41 smithi183 ceph-mon[25285]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-20T08:41:41.705 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:41:41 smithi183 ceph-mon[25285]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-20T08:41:41.705 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:41:41 smithi183 ceph-mon[25285]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-20T08:41:41.705 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:41:41 smithi183 ceph-mon[25285]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-20T08:41:41.705 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:41:41 smithi183 ceph-mon[25285]: mon.a is new leader, mons a,b in quorum (ranks 0,1) 2024-03-20T08:41:41.705 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:41:41 smithi183 ceph-mon[25285]: monmap e2: 2 mons at {a=[v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0],b=[v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-20T08:41:41.705 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:41:41 smithi183 ceph-mon[25285]: fsmap 2024-03-20T08:41:41.705 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:41:41 smithi183 ceph-mon[25285]: osdmap e4: 0 total, 0 up, 0 in 2024-03-20T08:41:41.705 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:41:41 smithi183 ceph-mon[25285]: mgrmap e13: y(active, since 41s) 2024-03-20T08:41:41.705 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:41:41 smithi183 ceph-mon[25285]: overall HEALTH_OK 2024-03-20T08:41:41.706 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:41:41 smithi183 ceph-mon[25285]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:41:41.706 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:41:41 smithi183 ceph-mon[25285]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:41:41.706 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:41:41 smithi183 ceph-mon[25285]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:41:41.706 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:41:41 smithi183 ceph-mon[25285]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-20T08:41:41.706 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:41:41 smithi183 ceph-mon[25285]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-20T08:41:42.638 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:41:42 smithi194 ceph-mon[27237]: Deploying daemon mon.c on smithi183 2024-03-20T08:41:42.638 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:41:42 smithi194 ceph-mon[27237]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-20T08:41:42.638 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:41:42 smithi194 ceph-mon[27237]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-20T08:41:42.704 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:41:42 smithi183 ceph-mon[25285]: Deploying daemon mon.c on smithi183 2024-03-20T08:41:42.704 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:41:42 smithi183 ceph-mon[25285]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-20T08:41:42.704 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:41:42 smithi183 ceph-mon[25285]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-20T08:41:42.855 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-03-20T08:41:42.856 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4c3f3eba-e695-11ee-95c9-87774f69a715 -- ceph mon dump -f json 2024-03-20T08:41:43.098 INFO:teuthology.orchestra.run.smithi194.stderr:Inferring config /var/lib/ceph/4c3f3eba-e695-11ee-95c9-87774f69a715/mon.b/config 2024-03-20T08:41:44.455 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:41:44 smithi183 systemd[1]: Starting Ceph mon.c for 4c3f3eba-e695-11ee-95c9-87774f69a715... 2024-03-20T08:41:44.686 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-03-20T08:41:44.686 INFO:teuthology.orchestra.run.smithi194.stdout:{"epoch":2,"fsid":"4c3f3eba-e695-11ee-95c9-87774f69a715","modified":"2024-03-20T08:41:36.184066Z","created":"2024-03-20T08:39:56.427572Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:3300","nonce":0},{"type":"v1","addr":"172.21.15.183:6789","nonce":0}]},"addr":"172.21.15.183:6789/0","public_addr":"172.21.15.183:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:3300","nonce":0},{"type":"v1","addr":"172.21.15.194:6789","nonce":0}]},"addr":"172.21.15.194:6789/0","public_addr":"172.21.15.194:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2024-03-20T08:41:44.688 INFO:teuthology.orchestra.run.smithi194.stderr:dumped monmap epoch 2 2024-03-20T08:41:44.815 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:41:44 smithi183 ceph-mon[25285]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-20T08:41:44.815 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:41:44 smithi183 podman[31840]: 2024-03-20 08:41:44.469733895 +0000 UTC m=+0.017320235 image pull 9ceba81fb2c8d8682cb159527017d591637b153b3aa4a390ffa8586db6dcda00 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:976a8b2d06e3c6bfd12488297e8584dd704cc06c3e976c7e0f718c13e238e912 2024-03-20T08:41:44.816 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:41:44 smithi183 podman[31840]: 2024-03-20 08:41:44.584598488 +0000 UTC m=+0.132184819 container create c5f899930acf7b1491b16ce5109556156eee62aa6429bc68a88d5b004157b894 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:976a8b2d06e3c6bfd12488297e8584dd704cc06c3e976c7e0f718c13e238e912, name=ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-mon-c, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240307, GIT_CLEAN=True, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.vendor=CentOS, RELEASE=wip-yuri5-testing-2024-03-18-1144-e7e80a6, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-20T08:41:44.816 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:41:44 smithi183 podman[31840]: 2024-03-20 08:41:44.778586936 +0000 UTC m=+0.326173267 container init c5f899930acf7b1491b16ce5109556156eee62aa6429bc68a88d5b004157b894 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:976a8b2d06e3c6bfd12488297e8584dd704cc06c3e976c7e0f718c13e238e912, name=ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-mon-c, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240307, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, RELEASE=wip-yuri5-testing-2024-03-18-1144-e7e80a6, GIT_BRANCH=HEAD) 2024-03-20T08:41:44.816 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:41:44 smithi183 podman[31840]: 2024-03-20 08:41:44.781665436 +0000 UTC m=+0.329251767 container start c5f899930acf7b1491b16ce5109556156eee62aa6429bc68a88d5b004157b894 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:976a8b2d06e3c6bfd12488297e8584dd704cc06c3e976c7e0f718c13e238e912, name=ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-mon-c, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, ceph=True, RELEASE=wip-yuri5-testing-2024-03-18-1144-e7e80a6, org.label-schema.build-date=20240307) 2024-03-20T08:41:44.816 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:41:44 smithi183 ceph-mon[31860]: set uid:gid to 167:167 (ceph:ceph) 2024-03-20T08:41:44.816 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:41:44 smithi183 ceph-mon[31860]: ceph version 19.0.0-2153-ge7e80a6a (e7e80a6ac9849f0f23432d0d9832f096730308ae) squid (dev), process ceph-mon, pid 2 2024-03-20T08:41:44.816 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:41:44 smithi183 ceph-mon[31860]: pidfile_write: ignore empty --pid-file 2024-03-20T08:41:44.816 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:41:44 smithi183 ceph-mon[31860]: load: jerasure load: lrc 2024-03-20T08:41:44.816 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:41:44 smithi183 ceph-mon[31860]: rocksdb: RocksDB version: 7.9.2 2024-03-20T08:41:44.816 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:41:44 smithi183 ceph-mon[31860]: rocksdb: Git sha 0 2024-03-20T08:41:44.816 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:41:44 smithi183 ceph-mon[31860]: rocksdb: Compile date 2024-03-18 19:18:37 2024-03-20T08:41:44.816 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:41:44 smithi183 ceph-mon[31860]: rocksdb: DB SUMMARY 2024-03-20T08:41:44.816 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:41:44 smithi183 ceph-mon[31860]: rocksdb: DB Session ID: 5NTSK1B3TTXBGYYVL8NO 2024-03-20T08:41:44.816 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:41:44 smithi183 ceph-mon[31860]: rocksdb: CURRENT file: CURRENT 2024-03-20T08:41:44.816 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:41:44 smithi183 ceph-mon[31860]: rocksdb: IDENTITY file: IDENTITY 2024-03-20T08:41:44.817 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:41:44 smithi183 ceph-mon[31860]: rocksdb: MANIFEST file: MANIFEST-000005 size: 59 Bytes 2024-03-20T08:41:44.817 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:41:44 smithi183 ceph-mon[31860]: rocksdb: SST files in /var/lib/ceph/mon/ceph-c/store.db dir, Total Num: 0, files: 2024-03-20T08:41:44.817 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:41:44 smithi183 ceph-mon[31860]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-c/store.db: 000004.log size: 636 ; 2024-03-20T08:41:44.817 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:41:44 smithi183 ceph-mon[31860]: rocksdb: Options.error_if_exists: 0 2024-03-20T08:41:44.817 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:41:44 smithi183 ceph-mon[31860]: rocksdb: Options.create_if_missing: 0 2024-03-20T08:41:44.817 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:41:44 smithi183 ceph-mon[31860]: rocksdb: Options.paranoid_checks: 1 2024-03-20T08:41:44.817 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:41:44 smithi183 ceph-mon[31860]: rocksdb: Options.flush_verify_memtable_count: 1 2024-03-20T08:41:44.817 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:41:44 smithi183 ceph-mon[31860]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-03-20T08:41:44.817 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:41:44 smithi183 ceph-mon[31860]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-03-20T08:41:44.817 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:41:44 smithi183 ceph-mon[31860]: rocksdb: Options.env: 0x5617e56e3b20 2024-03-20T08:41:44.817 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:41:44 smithi183 ceph-mon[31860]: rocksdb: Options.fs: PosixFileSystem 2024-03-20T08:41:44.817 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:41:44 smithi183 ceph-mon[31860]: rocksdb: Options.info_log: 0x5617e626b920 2024-03-20T08:41:44.817 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:41:44 smithi183 ceph-mon[31860]: rocksdb: Options.max_file_opening_threads: 16 2024-03-20T08:41:44.817 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:41:44 smithi183 ceph-mon[31860]: rocksdb: Options.statistics: (nil) 2024-03-20T08:41:44.818 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:41:44 smithi183 ceph-mon[31860]: rocksdb: Options.use_fsync: 0 2024-03-20T08:41:44.818 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:41:44 smithi183 ceph-mon[31860]: rocksdb: Options.max_log_file_size: 0 2024-03-20T08:41:44.818 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:41:44 smithi183 ceph-mon[31860]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-03-20T08:41:44.818 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:41:44 smithi183 ceph-mon[31860]: rocksdb: Options.log_file_time_to_roll: 0 2024-03-20T08:41:44.818 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:41:44 smithi183 ceph-mon[31860]: rocksdb: Options.keep_log_file_num: 1000 2024-03-20T08:41:44.818 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:41:44 smithi183 ceph-mon[31860]: rocksdb: Options.recycle_log_file_num: 0 2024-03-20T08:41:44.818 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:41:44 smithi183 ceph-mon[31860]: rocksdb: Options.allow_fallocate: 1 2024-03-20T08:41:44.818 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:41:44 smithi183 ceph-mon[31860]: rocksdb: Options.allow_mmap_reads: 0 2024-03-20T08:41:44.818 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:41:44 smithi183 ceph-mon[31860]: rocksdb: Options.allow_mmap_writes: 0 2024-03-20T08:41:44.818 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:41:44 smithi183 ceph-mon[31860]: rocksdb: Options.use_direct_reads: 0 2024-03-20T08:41:44.818 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:41:44 smithi183 ceph-mon[31860]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-03-20T08:41:44.818 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:41:44 smithi183 ceph-mon[31860]: rocksdb: Options.create_missing_column_families: 0 2024-03-20T08:41:44.818 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:41:44 smithi183 ceph-mon[31860]: rocksdb: Options.db_log_dir: 2024-03-20T08:41:44.818 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:41:44 smithi183 ceph-mon[31860]: rocksdb: Options.wal_dir: 2024-03-20T08:41:44.819 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:41:44 smithi183 ceph-mon[31860]: rocksdb: Options.table_cache_numshardbits: 6 2024-03-20T08:41:44.819 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:41:44 smithi183 ceph-mon[31860]: rocksdb: Options.WAL_ttl_seconds: 0 2024-03-20T08:41:44.819 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:41:44 smithi183 ceph-mon[31860]: rocksdb: Options.WAL_size_limit_MB: 0 2024-03-20T08:41:44.819 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:41:44 smithi183 ceph-mon[31860]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-03-20T08:41:44.819 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:41:44 smithi183 ceph-mon[31860]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-03-20T08:41:44.819 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:41:44 smithi183 ceph-mon[31860]: rocksdb: Options.is_fd_close_on_exec: 1 2024-03-20T08:41:44.819 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:41:44 smithi183 ceph-mon[31860]: rocksdb: Options.advise_random_on_open: 1 2024-03-20T08:41:44.819 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:41:44 smithi183 ceph-mon[31860]: rocksdb: Options.db_write_buffer_size: 0 2024-03-20T08:41:44.819 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:41:44 smithi183 ceph-mon[31860]: rocksdb: Options.write_buffer_manager: 0x5617e623e1e0 2024-03-20T08:41:44.819 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:41:44 smithi183 ceph-mon[31860]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-03-20T08:41:44.819 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:41:44 smithi183 ceph-mon[31860]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-03-20T08:41:44.819 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:41:44 smithi183 ceph-mon[31860]: rocksdb: Options.use_adaptive_mutex: 0 2024-03-20T08:41:44.819 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:41:44 smithi183 ceph-mon[31860]: rocksdb: Options.rate_limiter: (nil) 2024-03-20T08:41:44.821 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:41:44 smithi183 ceph-mon[31860]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-03-20T08:41:44.821 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:41:44 smithi183 ceph-mon[31860]: rocksdb: Options.wal_recovery_mode: 2 2024-03-20T08:41:44.821 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:41:44 smithi183 ceph-mon[31860]: rocksdb: Options.enable_thread_tracking: 0 2024-03-20T08:41:44.821 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:41:44 smithi183 ceph-mon[31860]: rocksdb: Options.enable_pipelined_write: 0 2024-03-20T08:41:44.821 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:41:44 smithi183 ceph-mon[31860]: rocksdb: Options.unordered_write: 0 2024-03-20T08:41:44.821 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:41:44 smithi183 ceph-mon[31860]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-03-20T08:41:44.821 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:41:44 smithi183 ceph-mon[31860]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-03-20T08:41:44.821 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:41:44 smithi183 ceph-mon[31860]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-03-20T08:41:44.822 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:41:44 smithi183 ceph-mon[31860]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-03-20T08:41:44.822 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:41:44 smithi183 ceph-mon[31860]: rocksdb: Options.row_cache: None 2024-03-20T08:41:44.822 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:41:44 smithi183 ceph-mon[31860]: rocksdb: Options.wal_filter: None 2024-03-20T08:41:44.822 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:41:44 smithi183 ceph-mon[31860]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-03-20T08:41:44.822 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:41:44 smithi183 ceph-mon[31860]: rocksdb: Options.allow_ingest_behind: 0 2024-03-20T08:41:44.822 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:41:44 smithi183 ceph-mon[31860]: rocksdb: Options.two_write_queues: 0 2024-03-20T08:41:44.822 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:41:44 smithi183 ceph-mon[31860]: rocksdb: Options.manual_wal_flush: 0 2024-03-20T08:41:44.822 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:41:44 smithi183 ceph-mon[31860]: rocksdb: Options.wal_compression: 0 2024-03-20T08:41:44.822 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:41:44 smithi183 ceph-mon[31860]: rocksdb: Options.atomic_flush: 0 2024-03-20T08:41:44.822 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:41:44 smithi183 ceph-mon[31860]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-03-20T08:41:44.822 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:41:44 smithi183 ceph-mon[31860]: rocksdb: Options.persist_stats_to_disk: 0 2024-03-20T08:41:44.822 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:41:44 smithi183 ceph-mon[31860]: rocksdb: Options.write_dbid_to_manifest: 0 2024-03-20T08:41:44.822 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:41:44 smithi183 ceph-mon[31860]: rocksdb: Options.log_readahead_size: 0 2024-03-20T08:41:44.822 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:41:44 smithi183 ceph-mon[31860]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-03-20T08:41:44.823 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:41:44 smithi183 ceph-mon[31860]: rocksdb: Options.best_efforts_recovery: 0 2024-03-20T08:41:44.823 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:41:44 smithi183 ceph-mon[31860]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-03-20T08:41:44.823 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:41:44 smithi183 ceph-mon[31860]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-03-20T08:41:44.823 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:41:44 smithi183 ceph-mon[31860]: rocksdb: Options.allow_data_in_errors: 0 2024-03-20T08:41:44.823 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:41:44 smithi183 ceph-mon[31860]: rocksdb: Options.db_host_id: __hostname__ 2024-03-20T08:41:44.823 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:41:44 smithi183 ceph-mon[31860]: rocksdb: Options.enforce_single_del_contracts: true 2024-03-20T08:41:44.823 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:41:44 smithi183 ceph-mon[31860]: rocksdb: Options.max_background_jobs: 2 2024-03-20T08:41:44.823 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:41:44 smithi183 ceph-mon[31860]: rocksdb: Options.max_background_compactions: -1 2024-03-20T08:41:44.823 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:41:44 smithi183 ceph-mon[31860]: rocksdb: Options.max_subcompactions: 1 2024-03-20T08:41:44.823 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:41:44 smithi183 ceph-mon[31860]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-03-20T08:41:44.823 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:41:44 smithi183 ceph-mon[31860]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-03-20T08:41:44.823 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:41:44 smithi183 ceph-mon[31860]: rocksdb: Options.delayed_write_rate : 16777216 2024-03-20T08:41:44.823 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:41:44 smithi183 ceph-mon[31860]: rocksdb: Options.max_total_wal_size: 0 2024-03-20T08:41:44.823 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:41:44 smithi183 ceph-mon[31860]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-03-20T08:41:44.823 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:41:44 smithi183 ceph-mon[31860]: rocksdb: Options.stats_dump_period_sec: 600 2024-03-20T08:41:44.824 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:41:44 smithi183 ceph-mon[31860]: rocksdb: Options.stats_persist_period_sec: 600 2024-03-20T08:41:44.824 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:41:44 smithi183 ceph-mon[31860]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-03-20T08:41:44.824 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:41:44 smithi183 ceph-mon[31860]: rocksdb: Options.max_open_files: -1 2024-03-20T08:41:44.824 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:41:44 smithi183 ceph-mon[31860]: rocksdb: Options.bytes_per_sync: 0 2024-03-20T08:41:44.824 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:41:44 smithi183 ceph-mon[31860]: rocksdb: Options.wal_bytes_per_sync: 0 2024-03-20T08:41:44.824 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:41:44 smithi183 ceph-mon[31860]: rocksdb: Options.strict_bytes_per_sync: 0 2024-03-20T08:41:44.824 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:41:44 smithi183 ceph-mon[31860]: rocksdb: Options.compaction_readahead_size: 0 2024-03-20T08:41:44.824 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:41:44 smithi183 ceph-mon[31860]: rocksdb: Options.max_background_flushes: -1 2024-03-20T08:41:44.824 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:41:44 smithi183 ceph-mon[31860]: rocksdb: Compression algorithms supported: 2024-03-20T08:41:44.824 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:41:44 smithi183 ceph-mon[31860]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-03-20T08:41:44.824 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:41:44 smithi183 ceph-mon[31860]: rocksdb: kZSTD supported: 0 2024-03-20T08:41:44.824 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:41:44 smithi183 ceph-mon[31860]: rocksdb: kXpressCompression supported: 0 2024-03-20T08:41:44.824 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:41:44 smithi183 ceph-mon[31860]: rocksdb: kLZ4HCCompression supported: 1 2024-03-20T08:41:44.824 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:41:44 smithi183 ceph-mon[31860]: rocksdb: kZlibCompression supported: 1 2024-03-20T08:41:44.824 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:41:44 smithi183 ceph-mon[31860]: rocksdb: kSnappyCompression supported: 1 2024-03-20T08:41:44.825 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:41:44 smithi183 ceph-mon[31860]: rocksdb: kLZ4Compression supported: 1 2024-03-20T08:41:44.825 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:41:44 smithi183 ceph-mon[31860]: rocksdb: kBZip2Compression supported: 0 2024-03-20T08:41:44.825 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:41:44 smithi183 ceph-mon[31860]: rocksdb: Fast CRC32 supported: Supported on x86 2024-03-20T08:41:44.825 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:41:44 smithi183 ceph-mon[31860]: rocksdb: DMutex implementation: pthread_mutex_t 2024-03-20T08:41:44.825 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:41:44 smithi183 ceph-mon[31860]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-c/store.db/MANIFEST-000005 2024-03-20T08:41:44.825 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:41:44 smithi183 ceph-mon[31860]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-03-20T08:41:44.825 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:41:44 smithi183 ceph-mon[31860]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-03-20T08:41:44.825 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:41:44 smithi183 ceph-mon[31860]: rocksdb: Options.merge_operator: 2024-03-20T08:41:44.826 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:41:44 smithi183 ceph-mon[31860]: rocksdb: Options.compaction_filter: None 2024-03-20T08:41:44.826 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:41:44 smithi183 ceph-mon[31860]: rocksdb: Options.compaction_filter_factory: None 2024-03-20T08:41:44.826 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:41:44 smithi183 ceph-mon[31860]: rocksdb: Options.sst_partitioner_factory: None 2024-03-20T08:41:44.826 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:41:44 smithi183 ceph-mon[31860]: rocksdb: Options.memtable_factory: SkipListFactory 2024-03-20T08:41:44.826 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:41:44 smithi183 ceph-mon[31860]: rocksdb: Options.table_factory: BlockBasedTable 2024-03-20T08:41:44.826 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:41:44 smithi183 ceph-mon[31860]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x5617e626ba20) 2024-03-20T08:41:44.826 INFO:journalctl@ceph.mon.c.smithi183.stdout: cache_index_and_filter_blocks: 1 2024-03-20T08:41:44.826 INFO:journalctl@ceph.mon.c.smithi183.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-03-20T08:41:44.826 INFO:journalctl@ceph.mon.c.smithi183.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-03-20T08:41:44.827 INFO:journalctl@ceph.mon.c.smithi183.stdout: pin_top_level_index_and_filter: 1 2024-03-20T08:41:44.827 INFO:journalctl@ceph.mon.c.smithi183.stdout: index_type: 0 2024-03-20T08:41:44.827 INFO:journalctl@ceph.mon.c.smithi183.stdout: data_block_index_type: 0 2024-03-20T08:41:44.827 INFO:journalctl@ceph.mon.c.smithi183.stdout: index_shortening: 1 2024-03-20T08:41:44.827 INFO:journalctl@ceph.mon.c.smithi183.stdout: data_block_hash_table_util_ratio: 0.750000 2024-03-20T08:41:44.827 INFO:journalctl@ceph.mon.c.smithi183.stdout: checksum: 4 2024-03-20T08:41:44.827 INFO:journalctl@ceph.mon.c.smithi183.stdout: no_block_cache: 0 2024-03-20T08:41:44.827 INFO:journalctl@ceph.mon.c.smithi183.stdout: block_cache: 0x5617e6271090 2024-03-20T08:41:44.827 INFO:journalctl@ceph.mon.c.smithi183.stdout: block_cache_name: BinnedLRUCache 2024-03-20T08:41:44.827 INFO:journalctl@ceph.mon.c.smithi183.stdout: block_cache_options: 2024-03-20T08:41:44.827 INFO:journalctl@ceph.mon.c.smithi183.stdout: capacity : 536870912 2024-03-20T08:41:44.827 INFO:journalctl@ceph.mon.c.smithi183.stdout: num_shard_bits : 4 2024-03-20T08:41:44.827 INFO:journalctl@ceph.mon.c.smithi183.stdout: strict_capacity_limit : 0 2024-03-20T08:41:44.827 INFO:journalctl@ceph.mon.c.smithi183.stdout: high_pri_pool_ratio: 0.000 2024-03-20T08:41:44.827 INFO:journalctl@ceph.mon.c.smithi183.stdout: block_cache_compressed: (nil) 2024-03-20T08:41:44.828 INFO:journalctl@ceph.mon.c.smithi183.stdout: persistent_cache: (nil) 2024-03-20T08:41:44.828 INFO:journalctl@ceph.mon.c.smithi183.stdout: block_size: 4096 2024-03-20T08:41:44.828 INFO:journalctl@ceph.mon.c.smithi183.stdout: block_size_deviation: 10 2024-03-20T08:41:44.828 INFO:journalctl@ceph.mon.c.smithi183.stdout: block_restart_interval: 16 2024-03-20T08:41:44.828 INFO:journalctl@ceph.mon.c.smithi183.stdout: index_block_restart_interval: 1 2024-03-20T08:41:44.828 INFO:journalctl@ceph.mon.c.smithi183.stdout: metadata_block_size: 4096 2024-03-20T08:41:44.828 INFO:journalctl@ceph.mon.c.smithi183.stdout: partition_filters: 0 2024-03-20T08:41:44.828 INFO:journalctl@ceph.mon.c.smithi183.stdout: use_delta_encoding: 1 2024-03-20T08:41:44.828 INFO:journalctl@ceph.mon.c.smithi183.stdout: filter_policy: bloomfilter 2024-03-20T08:41:44.828 INFO:journalctl@ceph.mon.c.smithi183.stdout: whole_key_filtering: 1 2024-03-20T08:41:44.828 INFO:journalctl@ceph.mon.c.smithi183.stdout: verify_compression: 0 2024-03-20T08:41:44.828 INFO:journalctl@ceph.mon.c.smithi183.stdout: read_amp_bytes_per_bit: 0 2024-03-20T08:41:44.828 INFO:journalctl@ceph.mon.c.smithi183.stdout: format_version: 5 2024-03-20T08:41:44.828 INFO:journalctl@ceph.mon.c.smithi183.stdout: enable_index_compression: 1 2024-03-20T08:41:44.828 INFO:journalctl@ceph.mon.c.smithi183.stdout: block_align: 0 2024-03-20T08:41:44.829 INFO:journalctl@ceph.mon.c.smithi183.stdout: max_auto_readahead_size: 262144 2024-03-20T08:41:44.829 INFO:journalctl@ceph.mon.c.smithi183.stdout: prepopulate_block_cache: 0 2024-03-20T08:41:44.829 INFO:journalctl@ceph.mon.c.smithi183.stdout: initial_auto_readahead_size: 8192 2024-03-20T08:41:44.829 INFO:journalctl@ceph.mon.c.smithi183.stdout: num_file_reads_for_auto_readahead: 2 2024-03-20T08:41:44.829 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:41:44 smithi183 ceph-mon[31860]: rocksdb: Options.write_buffer_size: 33554432 2024-03-20T08:41:44.829 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:41:44 smithi183 ceph-mon[31860]: rocksdb: Options.max_write_buffer_number: 2 2024-03-20T08:41:44.829 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:41:44 smithi183 ceph-mon[31860]: rocksdb: Options.compression: NoCompression 2024-03-20T08:41:44.829 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:41:44 smithi183 ceph-mon[31860]: rocksdb: Options.bottommost_compression: Disabled 2024-03-20T08:41:44.829 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:41:44 smithi183 ceph-mon[31860]: rocksdb: Options.prefix_extractor: nullptr 2024-03-20T08:41:44.829 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:41:44 smithi183 ceph-mon[31860]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-03-20T08:41:44.829 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:41:44 smithi183 ceph-mon[31860]: rocksdb: Options.num_levels: 7 2024-03-20T08:41:44.829 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:41:44 smithi183 ceph-mon[31860]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-03-20T08:41:44.829 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:41:44 smithi183 ceph-mon[31860]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-03-20T08:41:44.829 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:41:44 smithi183 ceph-mon[31860]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-03-20T08:41:44.830 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:41:44 smithi183 ceph-mon[31860]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-03-20T08:41:44.830 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:41:44 smithi183 ceph-mon[31860]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-03-20T08:41:44.830 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:41:44 smithi183 ceph-mon[31860]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-03-20T08:41:44.830 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:41:44 smithi183 ceph-mon[31860]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-03-20T08:41:44.830 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:41:44 smithi183 ceph-mon[31860]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-03-20T08:41:44.830 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:41:44 smithi183 ceph-mon[31860]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-03-20T08:41:44.830 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:41:44 smithi183 ceph-mon[31860]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-03-20T08:41:44.830 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:41:44 smithi183 ceph-mon[31860]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-03-20T08:41:44.830 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:41:44 smithi183 ceph-mon[31860]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-03-20T08:41:44.830 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:41:44 smithi183 ceph-mon[31860]: rocksdb: Options.compression_opts.window_bits: -14 2024-03-20T08:41:44.830 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:41:44 smithi183 ceph-mon[31860]: rocksdb: Options.compression_opts.level: 32767 2024-03-20T08:41:44.830 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:41:44 smithi183 ceph-mon[31860]: rocksdb: Options.compression_opts.strategy: 0 2024-03-20T08:41:44.830 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:41:44 smithi183 ceph-mon[31860]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-03-20T08:41:44.830 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:41:44 smithi183 ceph-mon[31860]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-03-20T08:41:44.830 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:41:44 smithi183 ceph-mon[31860]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-03-20T08:41:44.831 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:41:44 smithi183 ceph-mon[31860]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-03-20T08:41:44.831 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:41:44 smithi183 ceph-mon[31860]: rocksdb: Options.compression_opts.enabled: false 2024-03-20T08:41:44.831 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:41:44 smithi183 ceph-mon[31860]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-03-20T08:41:44.831 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:41:44 smithi183 ceph-mon[31860]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-03-20T08:41:44.831 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:41:44 smithi183 ceph-mon[31860]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-03-20T08:41:44.831 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:41:44 smithi183 ceph-mon[31860]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-03-20T08:41:44.831 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:41:44 smithi183 ceph-mon[31860]: rocksdb: Options.target_file_size_base: 67108864 2024-03-20T08:41:44.831 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:41:44 smithi183 ceph-mon[31860]: rocksdb: Options.target_file_size_multiplier: 1 2024-03-20T08:41:44.831 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:41:44 smithi183 ceph-mon[31860]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-03-20T08:41:44.831 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:41:44 smithi183 ceph-mon[31860]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-03-20T08:41:44.831 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:41:44 smithi183 ceph-mon[31860]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-03-20T08:41:44.831 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:41:44 smithi183 ceph-mon[31860]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-03-20T08:41:44.831 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:41:44 smithi183 ceph-mon[31860]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-03-20T08:41:44.832 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:41:44 smithi183 ceph-mon[31860]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-03-20T08:41:44.832 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:41:44 smithi183 ceph-mon[31860]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-03-20T08:41:44.832 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:41:44 smithi183 ceph-mon[31860]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-03-20T08:41:44.832 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:41:44 smithi183 ceph-mon[31860]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-03-20T08:41:44.832 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:41:44 smithi183 ceph-mon[31860]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-03-20T08:41:44.832 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:41:44 smithi183 ceph-mon[31860]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-03-20T08:41:44.832 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:41:44 smithi183 ceph-mon[31860]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-03-20T08:41:44.832 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:41:44 smithi183 ceph-mon[31860]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-03-20T08:41:44.832 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:41:44 smithi183 ceph-mon[31860]: rocksdb: Options.arena_block_size: 1048576 2024-03-20T08:41:44.832 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:41:44 smithi183 ceph-mon[31860]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-03-20T08:41:44.832 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:41:44 smithi183 ceph-mon[31860]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-03-20T08:41:44.832 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:41:44 smithi183 ceph-mon[31860]: rocksdb: Options.disable_auto_compactions: 0 2024-03-20T08:41:44.832 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:41:44 smithi183 ceph-mon[31860]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-03-20T08:41:44.832 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:41:44 smithi183 ceph-mon[31860]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-03-20T08:41:44.832 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:41:44 smithi183 ceph-mon[31860]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-03-20T08:41:44.833 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:41:44 smithi183 ceph-mon[31860]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-03-20T08:41:44.833 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:41:44 smithi183 ceph-mon[31860]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-03-20T08:41:44.833 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:41:44 smithi183 ceph-mon[31860]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-03-20T08:41:44.833 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:41:44 smithi183 ceph-mon[31860]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-03-20T08:41:44.833 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:41:44 smithi183 ceph-mon[31860]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-03-20T08:41:44.833 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:41:44 smithi183 ceph-mon[31860]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-03-20T08:41:44.833 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:41:44 smithi183 ceph-mon[31860]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-03-20T08:41:44.833 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:41:44 smithi183 ceph-mon[31860]: rocksdb: Options.table_properties_collectors: 2024-03-20T08:41:44.833 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:41:44 smithi183 ceph-mon[31860]: rocksdb: Options.inplace_update_support: 0 2024-03-20T08:41:44.833 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:41:44 smithi183 ceph-mon[31860]: rocksdb: Options.inplace_update_num_locks: 10000 2024-03-20T08:41:44.833 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:41:44 smithi183 ceph-mon[31860]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-03-20T08:41:44.833 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:41:44 smithi183 ceph-mon[31860]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-03-20T08:41:44.833 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:41:44 smithi183 ceph-mon[31860]: rocksdb: Options.memtable_huge_page_size: 0 2024-03-20T08:41:44.833 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:41:44 smithi183 ceph-mon[31860]: rocksdb: Options.bloom_locality: 0 2024-03-20T08:41:44.833 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:41:44 smithi183 ceph-mon[31860]: rocksdb: Options.max_successive_merges: 0 2024-03-20T08:41:44.834 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:41:44 smithi183 ceph-mon[31860]: rocksdb: Options.optimize_filters_for_hits: 0 2024-03-20T08:41:44.834 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:41:44 smithi183 ceph-mon[31860]: rocksdb: Options.paranoid_file_checks: 0 2024-03-20T08:41:44.834 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:41:44 smithi183 ceph-mon[31860]: rocksdb: Options.force_consistency_checks: 1 2024-03-20T08:41:44.834 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:41:44 smithi183 ceph-mon[31860]: rocksdb: Options.report_bg_io_stats: 0 2024-03-20T08:41:44.834 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:41:44 smithi183 ceph-mon[31860]: rocksdb: Options.ttl: 2592000 2024-03-20T08:41:44.834 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:41:44 smithi183 ceph-mon[31860]: rocksdb: Options.periodic_compaction_seconds: 0 2024-03-20T08:41:44.834 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:41:44 smithi183 ceph-mon[31860]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-03-20T08:41:44.834 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:41:44 smithi183 ceph-mon[31860]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-03-20T08:41:44.834 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:41:44 smithi183 ceph-mon[31860]: rocksdb: Options.enable_blob_files: false 2024-03-20T08:41:44.834 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:41:44 smithi183 ceph-mon[31860]: rocksdb: Options.min_blob_size: 0 2024-03-20T08:41:44.834 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:41:44 smithi183 ceph-mon[31860]: rocksdb: Options.blob_file_size: 268435456 2024-03-20T08:41:44.834 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:41:44 smithi183 ceph-mon[31860]: rocksdb: Options.blob_compression_type: NoCompression 2024-03-20T08:41:44.834 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:41:44 smithi183 ceph-mon[31860]: rocksdb: Options.enable_blob_garbage_collection: false 2024-03-20T08:41:44.834 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:41:44 smithi183 ceph-mon[31860]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-03-20T08:41:44.835 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:41:44 smithi183 ceph-mon[31860]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-03-20T08:41:44.835 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:41:44 smithi183 ceph-mon[31860]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-03-20T08:41:44.835 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:41:44 smithi183 ceph-mon[31860]: rocksdb: Options.blob_file_starting_level: 0 2024-03-20T08:41:44.835 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:41:44 smithi183 ceph-mon[31860]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-03-20T08:41:44.835 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:41:44 smithi183 ceph-mon[31860]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-c/store.db/MANIFEST-000005 succeeded,manifest_file_number is 5, next_file_number is 7, last_sequence is 0, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-03-20T08:41:44.835 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:41:44 smithi183 ceph-mon[31860]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 0 2024-03-20T08:41:44.835 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:41:44 smithi183 ceph-mon[31860]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: d671fc70-9f1e-413f-9794-78a5b4d952f8 2024-03-20T08:41:44.835 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:41:44 smithi183 ceph-mon[31860]: rocksdb: EVENT_LOG_v1 {"time_micros": 1710924104811428, "job": 1, "event": "recovery_started", "wal_files": [4]} 2024-03-20T08:41:44.835 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:41:44 smithi183 ceph-mon[31860]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #4 mode 2 2024-03-20T08:41:44.835 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:41:44 smithi183 ceph-mon[31860]: rocksdb: EVENT_LOG_v1 {"time_micros": 1710924104812009, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 8, "file_size": 1747, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 1, "largest_seqno": 5, "table_properties": {"data_size": 648, "index_size": 31, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 69, "raw_key_size": 115, "raw_average_key_size": 23, "raw_value_size": 526, "raw_average_value_size": 105, "num_data_blocks": 1, "num_entries": 5, "num_filter_entries": 5, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1710924104, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d671fc70-9f1e-413f-9794-78a5b4d952f8", "db_session_id": "5NTSK1B3TTXBGYYVL8NO", "orig_file_number": 8, "seqno_to_time_mapping": "N/A"}} 2024-03-20T08:41:44.835 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:41:44 smithi183 ceph-mon[31860]: rocksdb: EVENT_LOG_v1 {"time_micros": 1710924104812117, "job": 1, "event": "recovery_finished"} 2024-03-20T08:41:44.835 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:41:44 smithi183 ceph-mon[31860]: rocksdb: [db/version_set.cc:5047] Creating manifest 10 2024-03-20T08:41:44.835 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:41:44 smithi183 ceph-mon[31860]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-c/store.db/000004.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-03-20T08:41:44.835 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:41:44 smithi183 ceph-mon[31860]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x5617e6364000 2024-03-20T08:41:44.836 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:41:44 smithi183 ceph-mon[31860]: rocksdb: DB pointer 0x5617e634e000 2024-03-20T08:41:44.836 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:41:44 smithi183 ceph-mon[31860]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-03-20T08:41:44.836 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:41:44 smithi183 ceph-mon[31860]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-03-20T08:41:44.836 INFO:journalctl@ceph.mon.c.smithi183.stdout: ** DB Stats ** 2024-03-20T08:41:44.836 INFO:journalctl@ceph.mon.c.smithi183.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-03-20T08:41:44.836 INFO:journalctl@ceph.mon.c.smithi183.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-03-20T08:41:44.836 INFO:journalctl@ceph.mon.c.smithi183.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-03-20T08:41:44.836 INFO:journalctl@ceph.mon.c.smithi183.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-03-20T08:41:44.836 INFO:journalctl@ceph.mon.c.smithi183.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-03-20T08:41:44.836 INFO:journalctl@ceph.mon.c.smithi183.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-03-20T08:41:44.836 INFO:journalctl@ceph.mon.c.smithi183.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-03-20T08:41:44.836 INFO:journalctl@ceph.mon.c.smithi183.stdout: 2024-03-20T08:41:44.836 INFO:journalctl@ceph.mon.c.smithi183.stdout: ** Compaction Stats [default] ** 2024-03-20T08:41:44.836 INFO:journalctl@ceph.mon.c.smithi183.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-03-20T08:41:44.836 INFO:journalctl@ceph.mon.c.smithi183.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-03-20T08:41:44.837 INFO:journalctl@ceph.mon.c.smithi183.stdout: L0 1/0 1.71 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 3.0 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-03-20T08:41:44.837 INFO:journalctl@ceph.mon.c.smithi183.stdout: Sum 1/0 1.71 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 3.0 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-03-20T08:41:44.837 INFO:journalctl@ceph.mon.c.smithi183.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 3.0 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-03-20T08:41:44.837 INFO:journalctl@ceph.mon.c.smithi183.stdout: 2024-03-20T08:41:44.837 INFO:journalctl@ceph.mon.c.smithi183.stdout: ** Compaction Stats [default] ** 2024-03-20T08:41:44.837 INFO:journalctl@ceph.mon.c.smithi183.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-03-20T08:41:44.837 INFO:journalctl@ceph.mon.c.smithi183.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-03-20T08:41:44.837 INFO:journalctl@ceph.mon.c.smithi183.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 3.0 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-03-20T08:41:44.837 INFO:journalctl@ceph.mon.c.smithi183.stdout: 2024-03-20T08:41:44.837 INFO:journalctl@ceph.mon.c.smithi183.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-03-20T08:41:44.837 INFO:journalctl@ceph.mon.c.smithi183.stdout: 2024-03-20T08:41:44.837 INFO:journalctl@ceph.mon.c.smithi183.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-03-20T08:41:44.837 INFO:journalctl@ceph.mon.c.smithi183.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-03-20T08:41:44.837 INFO:journalctl@ceph.mon.c.smithi183.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-03-20T08:41:44.837 INFO:journalctl@ceph.mon.c.smithi183.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-03-20T08:41:44.838 INFO:journalctl@ceph.mon.c.smithi183.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-03-20T08:41:44.838 INFO:journalctl@ceph.mon.c.smithi183.stdout: AddFile(Keys): cumulative 0, interval 0 2024-03-20T08:41:44.838 INFO:journalctl@ceph.mon.c.smithi183.stdout: Cumulative compaction: 0.00 GB write, 0.49 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-20T08:41:44.838 INFO:journalctl@ceph.mon.c.smithi183.stdout: Interval compaction: 0.00 GB write, 0.49 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-20T08:41:44.838 INFO:journalctl@ceph.mon.c.smithi183.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-03-20T08:41:44.838 INFO:journalctl@ceph.mon.c.smithi183.stdout: Block cache BinnedLRUCache@0x5617e6271090#2 capacity: 512.00 MB usage: 0.22 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 9e-06 secs_since: 0 2024-03-20T08:41:44.838 INFO:journalctl@ceph.mon.c.smithi183.stdout: Block cache entry stats(count,size,portion): FilterBlock(1,0.11 KB,2.08616e-05%) IndexBlock(1,0.11 KB,2.08616e-05%) Misc(1,0.00 KB,0%) 2024-03-20T08:41:44.838 INFO:journalctl@ceph.mon.c.smithi183.stdout: 2024-03-20T08:41:44.838 INFO:journalctl@ceph.mon.c.smithi183.stdout: ** File Read Latency Histogram By Level [default] ** 2024-03-20T08:41:44.838 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:41:44 smithi183 ceph-mon[31860]: mon.c does not exist in monmap, will attempt to join an existing cluster 2024-03-20T08:41:44.838 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:41:44 smithi183 ceph-mon[31860]: using public_addrv [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] 2024-03-20T08:41:44.838 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:41:44 smithi183 ceph-mon[31860]: starting mon.c rank -1 at public addrs [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] at bind addrs [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] mon_data /var/lib/ceph/mon/ceph-c fsid 4c3f3eba-e695-11ee-95c9-87774f69a715 2024-03-20T08:41:44.838 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:41:44 smithi183 ceph-mon[31860]: mon.c@-1(???) e0 preinit fsid 4c3f3eba-e695-11ee-95c9-87774f69a715 2024-03-20T08:41:44.888 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:41:44 smithi194 ceph-mon[27237]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-20T08:41:45.205 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:41:44 smithi183 ceph-mon[31860]: mon.c@-1(synchronizing).mds e1 new map 2024-03-20T08:41:45.205 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:41:44 smithi183 ceph-mon[31860]: mon.c@-1(synchronizing).mds e1 print_map 2024-03-20T08:41:45.205 INFO:journalctl@ceph.mon.c.smithi183.stdout: e1 2024-03-20T08:41:45.205 INFO:journalctl@ceph.mon.c.smithi183.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-03-20T08:41:45.205 INFO:journalctl@ceph.mon.c.smithi183.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments} 2024-03-20T08:41:45.205 INFO:journalctl@ceph.mon.c.smithi183.stdout: legacy client fscid: -1 2024-03-20T08:41:45.205 INFO:journalctl@ceph.mon.c.smithi183.stdout: 2024-03-20T08:41:45.205 INFO:journalctl@ceph.mon.c.smithi183.stdout: No filesystems configured 2024-03-20T08:41:45.205 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:41:44 smithi183 ceph-mon[31860]: mon.c@-1(synchronizing).osd e0 _set_cache_ratios kv ratio 0.25 inc ratio 0.375 full ratio 0.375 2024-03-20T08:41:45.205 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:41:44 smithi183 ceph-mon[31860]: mon.c@-1(synchronizing).osd e0 register_cache_with_pcm pcm target: 2147483648 pcm max: 1020054732 pcm min: 134217728 inc_osd_cache size: 1 2024-03-20T08:41:45.205 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:41:44 smithi183 ceph-mon[31860]: mon.c@-1(synchronizing).osd e1 e1: 0 total, 0 up, 0 in 2024-03-20T08:41:45.205 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:41:44 smithi183 ceph-mon[31860]: mon.c@-1(synchronizing).osd e2 e2: 0 total, 0 up, 0 in 2024-03-20T08:41:45.206 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:41:44 smithi183 ceph-mon[31860]: mon.c@-1(synchronizing).osd e3 e3: 0 total, 0 up, 0 in 2024-03-20T08:41:45.206 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:41:44 smithi183 ceph-mon[31860]: mon.c@-1(synchronizing).osd e4 e4: 0 total, 0 up, 0 in 2024-03-20T08:41:45.206 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:41:44 smithi183 ceph-mon[31860]: mon.c@-1(synchronizing).osd e4 crush map has features 3314932999778484224, adjusting msgr requires 2024-03-20T08:41:45.206 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:41:44 smithi183 ceph-mon[31860]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-03-20T08:41:45.206 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:41:44 smithi183 ceph-mon[31860]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-03-20T08:41:45.206 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:41:44 smithi183 ceph-mon[31860]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-03-20T08:41:45.206 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:41:44 smithi183 ceph-mon[31860]: Deploying cephadm binary to smithi194 2024-03-20T08:41:45.206 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:41:44 smithi183 ceph-mon[31860]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-20T08:41:45.206 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:41:44 smithi183 ceph-mon[31860]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-20T08:41:45.206 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:41:44 smithi183 ceph-mon[31860]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:41:45.206 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:41:44 smithi183 ceph-mon[31860]: Added host smithi194 2024-03-20T08:41:45.206 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:41:44 smithi183 ceph-mon[31860]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-20T08:41:45.206 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:41:44 smithi183 ceph-mon[31860]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:41:45.206 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:41:44 smithi183 ceph-mon[31860]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-20T08:41:45.206 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:41:44 smithi183 ceph-mon[31860]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:41:45.207 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:41:44 smithi183 ceph-mon[31860]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:41:45.207 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:41:44 smithi183 ceph-mon[31860]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi194", "name": "osd_memory_target"}]: dispatch 2024-03-20T08:41:45.207 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:41:44 smithi183 ceph-mon[31860]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-20T08:41:45.207 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:41:44 smithi183 ceph-mon[31860]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-20T08:41:45.207 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:41:44 smithi183 ceph-mon[31860]: Updating smithi194:/etc/ceph/ceph.conf 2024-03-20T08:41:45.207 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:41:44 smithi183 ceph-mon[31860]: Updating smithi194:/var/lib/ceph/4c3f3eba-e695-11ee-95c9-87774f69a715/config/ceph.conf 2024-03-20T08:41:45.207 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:41:44 smithi183 ceph-mon[31860]: Updating smithi194:/etc/ceph/ceph.client.admin.keyring 2024-03-20T08:41:45.207 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:41:44 smithi183 ceph-mon[31860]: Updating smithi194:/var/lib/ceph/4c3f3eba-e695-11ee-95c9-87774f69a715/config/ceph.client.admin.keyring 2024-03-20T08:41:45.207 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:41:44 smithi183 ceph-mon[31860]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:41:45.207 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:41:44 smithi183 ceph-mon[31860]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:41:45.207 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:41:44 smithi183 ceph-mon[31860]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.agent.smithi194", "caps": []}]: dispatch 2024-03-20T08:41:45.207 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:41:44 smithi183 ceph-mon[31860]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.agent.smithi194", "caps": []}]': finished 2024-03-20T08:41:45.207 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:41:44 smithi183 ceph-mon[31860]: Deploying daemon agent.smithi194 on smithi194 2024-03-20T08:41:45.207 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:41:44 smithi183 ceph-mon[31860]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-20T08:41:45.207 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:41:44 smithi183 ceph-mon[31860]: from='client.14180 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-20T08:41:45.208 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:41:44 smithi183 ceph-mon[31860]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:41:45.208 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:41:44 smithi183 ceph-mon[31860]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:41:45.208 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:41:44 smithi183 ceph-mon[31860]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:41:45.208 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:41:44 smithi183 ceph-mon[31860]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-20T08:41:45.208 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:41:44 smithi183 ceph-mon[31860]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-20T08:41:45.208 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:41:44 smithi183 ceph-mon[31860]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-20T08:41:45.208 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:41:44 smithi183 ceph-mon[31860]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:41:45.208 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:41:44 smithi183 ceph-mon[31860]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-20T08:41:45.208 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:41:44 smithi183 ceph-mon[31860]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-20T08:41:45.208 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:41:44 smithi183 ceph-mon[31860]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-20T08:41:45.208 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:41:44 smithi183 ceph-mon[31860]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:41:45.208 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:41:44 smithi183 ceph-mon[31860]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:41:45.208 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:41:44 smithi183 ceph-mon[31860]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:41:45.208 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:41:44 smithi183 ceph-mon[31860]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:41:45.209 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:41:44 smithi183 ceph-mon[31860]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-20T08:41:45.209 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:41:44 smithi183 ceph-mon[31860]: from='client.? 172.21.15.183:0/21530245' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-03-20T08:41:45.209 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:41:44 smithi183 ceph-mon[31860]: from='client.? 172.21.15.183:0/21530245' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-03-20T08:41:45.209 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:41:44 smithi183 ceph-mon[31860]: osdmap e4: 0 total, 0 up, 0 in 2024-03-20T08:41:45.209 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:41:44 smithi183 ceph-mon[31860]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:41:45.209 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:41:44 smithi183 ceph-mon[31860]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:41:45.209 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:41:44 smithi183 ceph-mon[31860]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-20T08:41:45.209 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:41:44 smithi183 ceph-mon[31860]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:41:45.209 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:41:44 smithi183 ceph-mon[31860]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:41:45.209 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:41:44 smithi183 ceph-mon[31860]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:41:45.209 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:41:44 smithi183 ceph-mon[31860]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:41:45.210 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:41:44 smithi183 ceph-mon[31860]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-20T08:41:45.210 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:41:44 smithi183 ceph-mon[31860]: from='client.14184 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi183:172.21.15.183=a;smithi183:[v2:172.21.15.183:3301,v1:172.21.15.183:6790]=c;smithi194:172.21.15.194=b", "target": ["mon-mgr", ""]}]: dispatch 2024-03-20T08:41:45.210 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:41:44 smithi183 ceph-mon[31860]: Saving service mon spec with placement smithi183:172.21.15.183=a;smithi183:[v2:172.21.15.183:3301,v1:172.21.15.183:6790]=c;smithi194:172.21.15.194=b;count:3 2024-03-20T08:41:45.210 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:41:44 smithi183 ceph-mon[31860]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:41:45.210 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:41:44 smithi183 ceph-mon[31860]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-20T08:41:45.210 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:41:44 smithi183 ceph-mon[31860]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-20T08:41:45.210 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:41:44 smithi183 ceph-mon[31860]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-20T08:41:45.210 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:41:44 smithi183 ceph-mon[31860]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:41:45.210 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:41:44 smithi183 ceph-mon[31860]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-20T08:41:45.210 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:41:44 smithi183 ceph-mon[31860]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-20T08:41:45.210 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:41:44 smithi183 ceph-mon[31860]: Deploying daemon mon.b on smithi194 2024-03-20T08:41:45.210 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:41:44 smithi183 ceph-mon[31860]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-20T08:41:45.211 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:41:44 smithi183 ceph-mon[31860]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-20T08:41:45.211 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:41:44 smithi183 ceph-mon[31860]: mon.a calling monitor election 2024-03-20T08:41:45.211 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:41:44 smithi183 ceph-mon[31860]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-20T08:41:45.211 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:41:44 smithi183 ceph-mon[31860]: from='client.? 172.21.15.194:0/1341410218' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-20T08:41:45.211 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:41:44 smithi183 ceph-mon[31860]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-20T08:41:45.211 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:41:44 smithi183 ceph-mon[31860]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-20T08:41:45.211 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:41:44 smithi183 ceph-mon[31860]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-20T08:41:45.211 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:41:44 smithi183 ceph-mon[31860]: mon.b calling monitor election 2024-03-20T08:41:45.211 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:41:44 smithi183 ceph-mon[31860]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-20T08:41:45.211 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:41:44 smithi183 ceph-mon[31860]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-20T08:41:45.211 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:41:44 smithi183 ceph-mon[31860]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-20T08:41:45.211 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:41:44 smithi183 ceph-mon[31860]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-20T08:41:45.211 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:41:44 smithi183 ceph-mon[31860]: mon.a is new leader, mons a,b in quorum (ranks 0,1) 2024-03-20T08:41:45.211 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:41:44 smithi183 ceph-mon[31860]: monmap e2: 2 mons at {a=[v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0],b=[v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-20T08:41:45.211 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:41:44 smithi183 ceph-mon[31860]: fsmap 2024-03-20T08:41:45.212 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:41:44 smithi183 ceph-mon[31860]: osdmap e4: 0 total, 0 up, 0 in 2024-03-20T08:41:45.212 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:41:44 smithi183 ceph-mon[31860]: mgrmap e13: y(active, since 41s) 2024-03-20T08:41:45.212 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:41:44 smithi183 ceph-mon[31860]: overall HEALTH_OK 2024-03-20T08:41:45.212 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:41:44 smithi183 ceph-mon[31860]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:41:45.212 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:41:44 smithi183 ceph-mon[31860]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:41:45.212 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:41:44 smithi183 ceph-mon[31860]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:41:45.212 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:41:44 smithi183 ceph-mon[31860]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-20T08:41:45.212 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:41:44 smithi183 ceph-mon[31860]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-20T08:41:45.212 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:41:44 smithi183 ceph-mon[31860]: Deploying daemon mon.c on smithi183 2024-03-20T08:41:45.212 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:41:44 smithi183 ceph-mon[31860]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-20T08:41:45.212 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:41:44 smithi183 ceph-mon[31860]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-20T08:41:45.212 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:41:44 smithi183 ceph-mon[31860]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-20T08:41:45.212 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:41:44 smithi183 ceph-mon[31860]: mon.c@-1(synchronizing).paxosservice(auth 1..5) refresh upgraded, format 0 -> 3 2024-03-20T08:41:45.212 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:41:44 smithi183 bash[31840]: c5f899930acf7b1491b16ce5109556156eee62aa6429bc68a88d5b004157b894 2024-03-20T08:41:45.212 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:41:44 smithi183 systemd[1]: Started Ceph mon.c for 4c3f3eba-e695-11ee-95c9-87774f69a715. 2024-03-20T08:41:46.227 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-03-20T08:41:46.228 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4c3f3eba-e695-11ee-95c9-87774f69a715 -- ceph mon dump -f json 2024-03-20T08:41:46.477 INFO:teuthology.orchestra.run.smithi194.stderr:Inferring config /var/lib/ceph/4c3f3eba-e695-11ee-95c9-87774f69a715/mon.b/config 2024-03-20T08:41:49.955 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:41:49 smithi183 ceph-mon[31860]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-20T08:41:49.955 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:41:49 smithi183 ceph-mon[31860]: mon.a calling monitor election 2024-03-20T08:41:49.955 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:41:49 smithi183 ceph-mon[31860]: mon.b calling monitor election 2024-03-20T08:41:49.955 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:41:49 smithi183 ceph-mon[31860]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-20T08:41:49.955 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:41:49 smithi183 ceph-mon[31860]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-20T08:41:49.955 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:41:49 smithi183 ceph-mon[31860]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-20T08:41:49.955 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:41:49 smithi183 ceph-mon[31860]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-20T08:41:49.955 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:41:49 smithi183 ceph-mon[31860]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-20T08:41:49.955 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:41:49 smithi183 ceph-mon[31860]: mon.c calling monitor election 2024-03-20T08:41:49.955 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:41:49 smithi183 ceph-mon[31860]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-20T08:41:49.955 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:41:49 smithi183 ceph-mon[31860]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-20T08:41:49.955 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:41:49 smithi183 ceph-mon[31860]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-20T08:41:49.956 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:41:49 smithi183 ceph-mon[31860]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-20T08:41:49.956 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:41:49 smithi183 ceph-mon[31860]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-20T08:41:49.956 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:41:49 smithi183 ceph-mon[31860]: mon.a calling monitor election 2024-03-20T08:41:49.956 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:41:49 smithi183 ceph-mon[31860]: mon.a is new leader, mons a,b,c in quorum (ranks 0,1,2) 2024-03-20T08:41:49.956 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:41:49 smithi183 ceph-mon[31860]: monmap e3: 3 mons at {a=[v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0],b=[v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0],c=[v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-20T08:41:49.956 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:41:49 smithi183 ceph-mon[31860]: fsmap 2024-03-20T08:41:49.956 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:41:49 smithi183 ceph-mon[31860]: osdmap e4: 0 total, 0 up, 0 in 2024-03-20T08:41:49.956 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:41:49 smithi183 ceph-mon[31860]: mgrmap e13: y(active, since 50s) 2024-03-20T08:41:49.956 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:41:49 smithi183 ceph-mon[31860]: overall HEALTH_OK 2024-03-20T08:41:49.956 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:41:49 smithi183 ceph-mon[31860]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:41:49.956 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:41:49 smithi183 ceph-mon[31860]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:41:49.956 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:41:49 smithi183 ceph-mon[31860]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:41:49.956 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:41:49 smithi183 ceph-mon[31860]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:41:49.956 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:41:49 smithi183 ceph-mon[31860]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:41:49.957 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:41:49 smithi183 ceph-mon[25285]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-20T08:41:49.957 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:41:49 smithi183 ceph-mon[25285]: mon.a calling monitor election 2024-03-20T08:41:49.957 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:41:49 smithi183 ceph-mon[25285]: mon.b calling monitor election 2024-03-20T08:41:49.957 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:41:49 smithi183 ceph-mon[25285]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-20T08:41:49.957 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:41:49 smithi183 ceph-mon[25285]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-20T08:41:49.957 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:41:49 smithi183 ceph-mon[25285]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-20T08:41:49.957 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:41:49 smithi183 ceph-mon[25285]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-20T08:41:49.957 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:41:49 smithi183 ceph-mon[25285]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-20T08:41:49.957 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:41:49 smithi183 ceph-mon[25285]: mon.c calling monitor election 2024-03-20T08:41:49.957 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:41:49 smithi183 ceph-mon[25285]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-20T08:41:49.957 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:41:49 smithi183 ceph-mon[25285]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-20T08:41:49.958 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:41:49 smithi183 ceph-mon[25285]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-20T08:41:49.958 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:41:49 smithi183 ceph-mon[25285]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-20T08:41:49.958 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:41:49 smithi183 ceph-mon[25285]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-20T08:41:49.958 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:41:49 smithi183 ceph-mon[25285]: mon.a calling monitor election 2024-03-20T08:41:49.958 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:41:49 smithi183 ceph-mon[25285]: mon.a is new leader, mons a,b,c in quorum (ranks 0,1,2) 2024-03-20T08:41:49.958 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:41:49 smithi183 ceph-mon[25285]: monmap e3: 3 mons at {a=[v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0],b=[v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0],c=[v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-20T08:41:49.958 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:41:49 smithi183 ceph-mon[25285]: fsmap 2024-03-20T08:41:49.958 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:41:49 smithi183 ceph-mon[25285]: osdmap e4: 0 total, 0 up, 0 in 2024-03-20T08:41:49.958 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:41:49 smithi183 ceph-mon[25285]: mgrmap e13: y(active, since 50s) 2024-03-20T08:41:49.958 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:41:49 smithi183 ceph-mon[25285]: overall HEALTH_OK 2024-03-20T08:41:49.958 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:41:49 smithi183 ceph-mon[25285]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:41:49.958 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:41:49 smithi183 ceph-mon[25285]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:41:49.958 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:41:49 smithi183 ceph-mon[25285]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:41:49.959 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:41:49 smithi183 ceph-mon[25285]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:41:49.959 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:41:49 smithi183 ceph-mon[25285]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:41:50.138 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:41:49 smithi194 ceph-mon[27237]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-20T08:41:50.139 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:41:49 smithi194 ceph-mon[27237]: mon.a calling monitor election 2024-03-20T08:41:50.139 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:41:49 smithi194 ceph-mon[27237]: mon.b calling monitor election 2024-03-20T08:41:50.139 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:41:49 smithi194 ceph-mon[27237]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-20T08:41:50.139 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:41:49 smithi194 ceph-mon[27237]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-20T08:41:50.139 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:41:49 smithi194 ceph-mon[27237]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-20T08:41:50.139 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:41:49 smithi194 ceph-mon[27237]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-20T08:41:50.139 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:41:49 smithi194 ceph-mon[27237]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-20T08:41:50.139 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:41:49 smithi194 ceph-mon[27237]: mon.c calling monitor election 2024-03-20T08:41:50.139 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:41:49 smithi194 ceph-mon[27237]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-20T08:41:50.139 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:41:49 smithi194 ceph-mon[27237]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-20T08:41:50.139 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:41:49 smithi194 ceph-mon[27237]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-20T08:41:50.139 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:41:49 smithi194 ceph-mon[27237]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-20T08:41:50.140 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:41:49 smithi194 ceph-mon[27237]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-20T08:41:50.140 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:41:49 smithi194 ceph-mon[27237]: mon.a calling monitor election 2024-03-20T08:41:50.140 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:41:49 smithi194 ceph-mon[27237]: mon.a is new leader, mons a,b,c in quorum (ranks 0,1,2) 2024-03-20T08:41:50.140 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:41:49 smithi194 ceph-mon[27237]: monmap e3: 3 mons at {a=[v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0],b=[v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0],c=[v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-20T08:41:50.140 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:41:49 smithi194 ceph-mon[27237]: fsmap 2024-03-20T08:41:50.140 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:41:49 smithi194 ceph-mon[27237]: osdmap e4: 0 total, 0 up, 0 in 2024-03-20T08:41:50.140 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:41:49 smithi194 ceph-mon[27237]: mgrmap e13: y(active, since 50s) 2024-03-20T08:41:50.140 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:41:49 smithi194 ceph-mon[27237]: overall HEALTH_OK 2024-03-20T08:41:50.140 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:41:49 smithi194 ceph-mon[27237]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:41:50.140 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:41:49 smithi194 ceph-mon[27237]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:41:50.140 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:41:49 smithi194 ceph-mon[27237]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:41:50.140 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:41:49 smithi194 ceph-mon[27237]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:41:50.140 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:41:49 smithi194 ceph-mon[27237]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:41:50.226 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-03-20T08:41:50.227 INFO:teuthology.orchestra.run.smithi194.stdout:{"epoch":3,"fsid":"4c3f3eba-e695-11ee-95c9-87774f69a715","modified":"2024-03-20T08:41:44.834724Z","created":"2024-03-20T08:39:56.427572Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:3300","nonce":0},{"type":"v1","addr":"172.21.15.183:6789","nonce":0}]},"addr":"172.21.15.183:6789/0","public_addr":"172.21.15.183:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:3300","nonce":0},{"type":"v1","addr":"172.21.15.194:6789","nonce":0}]},"addr":"172.21.15.194:6789/0","public_addr":"172.21.15.194:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:3301","nonce":0},{"type":"v1","addr":"172.21.15.183:6790","nonce":0}]},"addr":"172.21.15.183:6790/0","public_addr":"172.21.15.183:6790/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1,2]} 2024-03-20T08:41:50.229 INFO:teuthology.orchestra.run.smithi194.stderr:dumped monmap epoch 3 2024-03-20T08:41:51.030 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:41:50 smithi183 ceph-mon[31860]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:41:51.030 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:41:50 smithi183 ceph-mon[31860]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:41:51.030 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:41:50 smithi183 ceph-mon[31860]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-20T08:41:51.030 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:41:50 smithi183 ceph-mon[31860]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-20T08:41:51.030 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:41:50 smithi183 ceph-mon[31860]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-20T08:41:51.030 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:41:50 smithi183 ceph-mon[31860]: Updating smithi183:/etc/ceph/ceph.conf 2024-03-20T08:41:51.030 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:41:50 smithi183 ceph-mon[31860]: Updating smithi194:/etc/ceph/ceph.conf 2024-03-20T08:41:51.031 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:41:50 smithi183 ceph-mon[31860]: from='client.? 172.21.15.194:0/2045970654' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-20T08:41:51.031 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:41:50 smithi183 ceph-mon[31860]: Updating smithi194:/var/lib/ceph/4c3f3eba-e695-11ee-95c9-87774f69a715/config/ceph.conf 2024-03-20T08:41:51.031 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:41:50 smithi183 ceph-mon[31860]: Updating smithi183:/var/lib/ceph/4c3f3eba-e695-11ee-95c9-87774f69a715/config/ceph.conf 2024-03-20T08:41:51.031 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:41:50 smithi183 ceph-mon[31860]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:41:51.031 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:41:50 smithi183 ceph-mon[31860]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:41:51.031 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:41:50 smithi183 ceph-mon[31860]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:41:51.031 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:41:50 smithi183 ceph-mon[31860]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:41:51.031 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:41:50 smithi183 ceph-mon[31860]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-20T08:41:51.031 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:41:50 smithi183 ceph-mon[31860]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:41:51.031 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:41:50 smithi183 ceph-mon[31860]: Metadata not up to date on all hosts. Skipping non agent specs 2024-03-20T08:41:51.031 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:41:50 smithi183 ceph-mon[31860]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:41:51.031 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:41:50 smithi183 ceph-mon[31860]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:41:51.031 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:41:50 smithi183 ceph-mon[31860]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:41:51.031 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:41:50 smithi183 ceph-mon[31860]: Reconfiguring mon.a (unknown last config time)... 2024-03-20T08:41:51.032 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:41:50 smithi183 ceph-mon[31860]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-20T08:41:51.032 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:41:50 smithi183 ceph-mon[31860]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-20T08:41:51.032 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:41:50 smithi183 ceph-mon[31860]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-20T08:41:51.032 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:41:50 smithi183 ceph-mon[31860]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-20T08:41:51.032 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:41:50 smithi183 ceph-mon[25285]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:41:51.032 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:41:50 smithi183 ceph-mon[25285]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:41:51.032 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:41:50 smithi183 ceph-mon[25285]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-20T08:41:51.032 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:41:50 smithi183 ceph-mon[25285]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-20T08:41:51.032 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:41:50 smithi183 ceph-mon[25285]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-20T08:41:51.032 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:41:50 smithi183 ceph-mon[25285]: Updating smithi183:/etc/ceph/ceph.conf 2024-03-20T08:41:51.032 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:41:50 smithi183 ceph-mon[25285]: Updating smithi194:/etc/ceph/ceph.conf 2024-03-20T08:41:51.032 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:41:50 smithi183 ceph-mon[25285]: from='client.? 172.21.15.194:0/2045970654' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-20T08:41:51.033 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:41:50 smithi183 ceph-mon[25285]: Updating smithi194:/var/lib/ceph/4c3f3eba-e695-11ee-95c9-87774f69a715/config/ceph.conf 2024-03-20T08:41:51.033 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:41:50 smithi183 ceph-mon[25285]: Updating smithi183:/var/lib/ceph/4c3f3eba-e695-11ee-95c9-87774f69a715/config/ceph.conf 2024-03-20T08:41:51.033 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:41:50 smithi183 ceph-mon[25285]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:41:51.033 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:41:50 smithi183 ceph-mon[25285]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:41:51.033 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:41:50 smithi183 ceph-mon[25285]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:41:51.033 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:41:50 smithi183 ceph-mon[25285]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:41:51.033 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:41:50 smithi183 ceph-mon[25285]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-20T08:41:51.033 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:41:50 smithi183 ceph-mon[25285]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:41:51.033 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:41:50 smithi183 ceph-mon[25285]: Metadata not up to date on all hosts. Skipping non agent specs 2024-03-20T08:41:51.033 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:41:50 smithi183 ceph-mon[25285]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:41:51.033 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:41:50 smithi183 ceph-mon[25285]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:41:51.033 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:41:50 smithi183 ceph-mon[25285]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:41:51.033 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:41:50 smithi183 ceph-mon[25285]: Reconfiguring mon.a (unknown last config time)... 2024-03-20T08:41:51.033 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:41:50 smithi183 ceph-mon[25285]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-20T08:41:51.034 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:41:50 smithi183 ceph-mon[25285]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-20T08:41:51.034 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:41:50 smithi183 ceph-mon[25285]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-20T08:41:51.034 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:41:50 smithi183 ceph-mon[25285]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-20T08:41:51.110 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:41:50 smithi194 ceph-mon[27237]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:41:51.111 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:41:50 smithi194 ceph-mon[27237]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:41:51.111 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:41:50 smithi194 ceph-mon[27237]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-20T08:41:51.111 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:41:50 smithi194 ceph-mon[27237]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-20T08:41:51.111 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:41:50 smithi194 ceph-mon[27237]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-20T08:41:51.111 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:41:50 smithi194 ceph-mon[27237]: Updating smithi183:/etc/ceph/ceph.conf 2024-03-20T08:41:51.111 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:41:50 smithi194 ceph-mon[27237]: Updating smithi194:/etc/ceph/ceph.conf 2024-03-20T08:41:51.111 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:41:50 smithi194 ceph-mon[27237]: from='client.? 172.21.15.194:0/2045970654' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-20T08:41:51.111 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:41:50 smithi194 ceph-mon[27237]: Updating smithi194:/var/lib/ceph/4c3f3eba-e695-11ee-95c9-87774f69a715/config/ceph.conf 2024-03-20T08:41:51.111 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:41:50 smithi194 ceph-mon[27237]: Updating smithi183:/var/lib/ceph/4c3f3eba-e695-11ee-95c9-87774f69a715/config/ceph.conf 2024-03-20T08:41:51.111 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:41:50 smithi194 ceph-mon[27237]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:41:51.111 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:41:50 smithi194 ceph-mon[27237]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:41:51.111 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:41:50 smithi194 ceph-mon[27237]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:41:51.112 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:41:50 smithi194 ceph-mon[27237]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:41:51.112 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:41:50 smithi194 ceph-mon[27237]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-20T08:41:51.112 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:41:50 smithi194 ceph-mon[27237]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:41:51.112 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:41:50 smithi194 ceph-mon[27237]: Metadata not up to date on all hosts. Skipping non agent specs 2024-03-20T08:41:51.112 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:41:50 smithi194 ceph-mon[27237]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:41:51.112 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:41:50 smithi194 ceph-mon[27237]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:41:51.112 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:41:50 smithi194 ceph-mon[27237]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:41:51.112 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:41:50 smithi194 ceph-mon[27237]: Reconfiguring mon.a (unknown last config time)... 2024-03-20T08:41:51.112 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:41:50 smithi194 ceph-mon[27237]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-20T08:41:51.112 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:41:50 smithi194 ceph-mon[27237]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-20T08:41:51.112 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:41:50 smithi194 ceph-mon[27237]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-20T08:41:51.113 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:41:50 smithi194 ceph-mon[27237]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-20T08:41:52.086 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:41:52 smithi194 ceph-mon[27237]: Reconfiguring daemon mon.a on smithi183 2024-03-20T08:41:52.086 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:41:52 smithi194 ceph-mon[27237]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:41:52.086 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:41:52 smithi194 ceph-mon[27237]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:41:52.086 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:41:52 smithi194 ceph-mon[27237]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:41:52.086 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:41:52 smithi194 ceph-mon[27237]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:41:52.086 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:41:52 smithi194 ceph-mon[27237]: Health check failed: 1 stray daemon(s) not managed by cephadm (CEPHADM_STRAY_DAEMON) 2024-03-20T08:41:52.204 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:41:52 smithi183 ceph-mon[25285]: Reconfiguring daemon mon.a on smithi183 2024-03-20T08:41:52.204 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:41:52 smithi183 ceph-mon[25285]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:41:52.204 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:41:52 smithi183 ceph-mon[25285]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:41:52.204 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:41:52 smithi183 ceph-mon[25285]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:41:52.205 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:41:52 smithi183 ceph-mon[25285]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:41:52.205 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:41:52 smithi183 ceph-mon[25285]: Health check failed: 1 stray daemon(s) not managed by cephadm (CEPHADM_STRAY_DAEMON) 2024-03-20T08:41:52.205 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:41:52 smithi183 ceph-mon[31860]: Reconfiguring daemon mon.a on smithi183 2024-03-20T08:41:52.205 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:41:52 smithi183 ceph-mon[31860]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:41:52.205 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:41:52 smithi183 ceph-mon[31860]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:41:52.205 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:41:52 smithi183 ceph-mon[31860]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:41:52.205 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:41:52 smithi183 ceph-mon[31860]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:41:52.205 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:41:52 smithi183 ceph-mon[31860]: Health check failed: 1 stray daemon(s) not managed by cephadm (CEPHADM_STRAY_DAEMON) 2024-03-20T08:41:52.726 INFO:tasks.cephadm:Generating final ceph.conf file... 2024-03-20T08:41:52.726 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4c3f3eba-e695-11ee-95c9-87774f69a715 -- ceph config generate-minimal-conf 2024-03-20T08:41:53.204 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:41:53 smithi183 ceph-mon[25285]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-20T08:41:53.204 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:41:53 smithi183 ceph-mon[31860]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-20T08:41:53.333 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:41:53 smithi194 ceph-mon[27237]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-20T08:41:54.351 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/4c3f3eba-e695-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T08:41:54.577 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:41:54 smithi183 ceph-mon[31860]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:41:54.577 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:41:54 smithi183 ceph-mon[31860]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:41:54.577 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:41:54 smithi183 ceph-mon[31860]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:41:54.577 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:41:54 smithi183 ceph-mon[31860]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:41:54.577 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:41:54 smithi183 ceph-mon[31860]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:41:54.578 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:41:54 smithi183 ceph-mon[31860]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:41:54.578 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:41:54 smithi183 ceph-mon[31860]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-20T08:41:54.578 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:41:54 smithi183 ceph-mon[31860]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-20T08:41:54.578 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:41:54 smithi183 ceph-mon[31860]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-20T08:41:54.578 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:41:54 smithi183 ceph-mon[31860]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:41:54.578 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:41:54 smithi183 ceph-mon[31860]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:41:54.578 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:41:54 smithi183 ceph-mon[31860]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:41:54.578 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:41:54 smithi183 ceph-mon[25285]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:41:54.578 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:41:54 smithi183 ceph-mon[25285]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:41:54.578 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:41:54 smithi183 ceph-mon[25285]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:41:54.578 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:41:54 smithi183 ceph-mon[25285]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:41:54.579 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:41:54 smithi183 ceph-mon[25285]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:41:54.579 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:41:54 smithi183 ceph-mon[25285]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:41:54.579 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:41:54 smithi183 ceph-mon[25285]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-20T08:41:54.579 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:41:54 smithi183 ceph-mon[25285]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-20T08:41:54.579 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:41:54 smithi183 ceph-mon[25285]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-20T08:41:54.579 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:41:54 smithi183 ceph-mon[25285]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:41:54.579 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:41:54 smithi183 ceph-mon[25285]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:41:54.579 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:41:54 smithi183 ceph-mon[25285]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:41:54.639 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:41:54 smithi194 ceph-mon[27237]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:41:54.639 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:41:54 smithi194 ceph-mon[27237]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:41:54.639 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:41:54 smithi194 ceph-mon[27237]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:41:54.639 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:41:54 smithi194 ceph-mon[27237]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:41:54.639 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:41:54 smithi194 ceph-mon[27237]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:41:54.639 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:41:54 smithi194 ceph-mon[27237]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:41:54.639 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:41:54 smithi194 ceph-mon[27237]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-20T08:41:54.639 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:41:54 smithi194 ceph-mon[27237]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-20T08:41:54.639 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:41:54 smithi194 ceph-mon[27237]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-20T08:41:54.639 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:41:54 smithi194 ceph-mon[27237]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:41:54.640 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:41:54 smithi194 ceph-mon[27237]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:41:54.640 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:41:54 smithi194 ceph-mon[27237]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:41:55.845 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:41:55 smithi194 ceph-mon[27237]: Reconfiguring mon.c (monmap changed)... 2024-03-20T08:41:55.845 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:41:55 smithi194 ceph-mon[27237]: Reconfiguring daemon mon.c on smithi183 2024-03-20T08:41:55.845 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:41:55 smithi194 ceph-mon[27237]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:41:55.845 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:41:55 smithi194 ceph-mon[27237]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-20T08:41:55.954 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:41:55 smithi183 ceph-mon[25285]: Reconfiguring mon.c (monmap changed)... 2024-03-20T08:41:55.954 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:41:55 smithi183 ceph-mon[25285]: Reconfiguring daemon mon.c on smithi183 2024-03-20T08:41:55.955 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:41:55 smithi183 ceph-mon[25285]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:41:55.955 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:41:55 smithi183 ceph-mon[25285]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-20T08:41:55.955 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:41:55 smithi183 ceph-mon[31860]: Reconfiguring mon.c (monmap changed)... 2024-03-20T08:41:55.955 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:41:55 smithi183 ceph-mon[31860]: Reconfiguring daemon mon.c on smithi183 2024-03-20T08:41:55.955 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:41:55 smithi183 ceph-mon[31860]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:41:55.955 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:41:55 smithi183 ceph-mon[31860]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-20T08:41:57.204 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:41:57 smithi183 ceph-mon[25285]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:41:57.204 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:41:57 smithi183 ceph-mon[25285]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:41:57.204 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:41:57 smithi183 ceph-mon[25285]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:41:57.204 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:41:57 smithi183 ceph-mon[25285]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-20T08:41:57.205 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:41:57 smithi183 ceph-mon[31860]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:41:57.205 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:41:57 smithi183 ceph-mon[31860]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:41:57.205 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:41:57 smithi183 ceph-mon[31860]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:41:57.205 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:41:57 smithi183 ceph-mon[31860]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-20T08:41:57.388 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:41:57 smithi194 ceph-mon[27237]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:41:57.388 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:41:57 smithi194 ceph-mon[27237]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:41:57.388 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:41:57 smithi194 ceph-mon[27237]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:41:57.388 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:41:57 smithi194 ceph-mon[27237]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-20T08:41:57.697 INFO:teuthology.orchestra.run.smithi183.stdout:# minimal ceph.conf for 4c3f3eba-e695-11ee-95c9-87774f69a715 2024-03-20T08:41:57.697 INFO:teuthology.orchestra.run.smithi183.stdout:[global] 2024-03-20T08:41:57.697 INFO:teuthology.orchestra.run.smithi183.stdout: fsid = 4c3f3eba-e695-11ee-95c9-87774f69a715 2024-03-20T08:41:57.697 INFO:teuthology.orchestra.run.smithi183.stdout: mon_host = [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] 2024-03-20T08:41:58.267 INFO:tasks.cephadm:Distributing (final) config and client.admin keyring... 2024-03-20T08:41:58.267 DEBUG:teuthology.orchestra.run.smithi183:> set -ex 2024-03-20T08:41:58.267 DEBUG:teuthology.orchestra.run.smithi183:> sudo dd of=/etc/ceph/ceph.conf 2024-03-20T08:41:58.299 DEBUG:teuthology.orchestra.run.smithi183:> set -ex 2024-03-20T08:41:58.299 DEBUG:teuthology.orchestra.run.smithi183:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-03-20T08:41:58.367 DEBUG:teuthology.orchestra.run.smithi194:> set -ex 2024-03-20T08:41:58.367 DEBUG:teuthology.orchestra.run.smithi194:> sudo dd of=/etc/ceph/ceph.conf 2024-03-20T08:41:58.400 DEBUG:teuthology.orchestra.run.smithi194:> set -ex 2024-03-20T08:41:58.400 DEBUG:teuthology.orchestra.run.smithi194:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-03-20T08:41:58.464 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:41:58 smithi194 ceph-mon[27237]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:41:58.464 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:41:58 smithi194 ceph-mon[27237]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:41:58.465 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:41:58 smithi194 ceph-mon[27237]: Reconfiguring mon.b (monmap changed)... 2024-03-20T08:41:58.465 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:41:58 smithi194 ceph-mon[27237]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-20T08:41:58.465 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:41:58 smithi194 ceph-mon[27237]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-20T08:41:58.465 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:41:58 smithi194 ceph-mon[27237]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-20T08:41:58.465 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:41:58 smithi194 ceph-mon[27237]: Reconfiguring daemon mon.b on smithi194 2024-03-20T08:41:58.465 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:41:58 smithi194 ceph-mon[27237]: from='client.? 172.21.15.183:0/3355644750' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-20T08:41:58.468 INFO:tasks.cephadm:Adding mgr.y on smithi183 2024-03-20T08:41:58.468 INFO:tasks.cephadm:Adding mgr.x on smithi194 2024-03-20T08:41:58.468 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4c3f3eba-e695-11ee-95c9-87774f69a715 -- ceph orch apply mgr '2;smithi183=y;smithi194=x' 2024-03-20T08:41:58.704 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:41:58 smithi183 ceph-mon[25285]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:41:58.704 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:41:58 smithi183 ceph-mon[25285]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:41:58.704 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:41:58 smithi183 ceph-mon[25285]: Reconfiguring mon.b (monmap changed)... 2024-03-20T08:41:58.704 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:41:58 smithi183 ceph-mon[25285]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-20T08:41:58.704 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:41:58 smithi183 ceph-mon[25285]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-20T08:41:58.705 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:41:58 smithi183 ceph-mon[25285]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-20T08:41:58.705 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:41:58 smithi183 ceph-mon[25285]: Reconfiguring daemon mon.b on smithi194 2024-03-20T08:41:58.705 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:41:58 smithi183 ceph-mon[25285]: from='client.? 172.21.15.183:0/3355644750' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-20T08:41:58.705 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:41:58 smithi183 ceph-mon[31860]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:41:58.705 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:41:58 smithi183 ceph-mon[31860]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:41:58.705 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:41:58 smithi183 ceph-mon[31860]: Reconfiguring mon.b (monmap changed)... 2024-03-20T08:41:58.705 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:41:58 smithi183 ceph-mon[31860]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-20T08:41:58.705 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:41:58 smithi183 ceph-mon[31860]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-20T08:41:58.705 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:41:58 smithi183 ceph-mon[31860]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-20T08:41:58.705 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:41:58 smithi183 ceph-mon[31860]: Reconfiguring daemon mon.b on smithi194 2024-03-20T08:41:58.705 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:41:58 smithi183 ceph-mon[31860]: from='client.? 172.21.15.183:0/3355644750' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-20T08:41:58.921 INFO:teuthology.orchestra.run.smithi194.stderr:Inferring config /var/lib/ceph/4c3f3eba-e695-11ee-95c9-87774f69a715/mon.b/config 2024-03-20T08:41:59.638 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:41:59 smithi194 ceph-mon[27237]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-20T08:41:59.638 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:41:59 smithi194 ceph-mon[27237]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:41:59.638 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:41:59 smithi194 ceph-mon[27237]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:41:59.638 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:41:59 smithi194 ceph-mon[27237]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-20T08:41:59.639 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:41:59 smithi194 ceph-mon[27237]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-20T08:41:59.639 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:41:59 smithi194 ceph-mon[27237]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-20T08:41:59.639 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:41:59 smithi194 ceph-mon[27237]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:41:59.704 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:41:59 smithi183 ceph-mon[25285]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-20T08:41:59.704 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:41:59 smithi183 ceph-mon[25285]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:41:59.704 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:41:59 smithi183 ceph-mon[25285]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:41:59.704 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:41:59 smithi183 ceph-mon[25285]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-20T08:41:59.705 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:41:59 smithi183 ceph-mon[25285]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-20T08:41:59.705 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:41:59 smithi183 ceph-mon[25285]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-20T08:41:59.705 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:41:59 smithi183 ceph-mon[25285]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:41:59.705 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:41:59 smithi183 ceph-mon[31860]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-20T08:41:59.705 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:41:59 smithi183 ceph-mon[31860]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:41:59.705 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:41:59 smithi183 ceph-mon[31860]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:41:59.705 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:41:59 smithi183 ceph-mon[31860]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-20T08:41:59.705 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:41:59 smithi183 ceph-mon[31860]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-20T08:41:59.705 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:41:59 smithi183 ceph-mon[31860]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-20T08:41:59.705 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:41:59 smithi183 ceph-mon[31860]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:42:00.484 INFO:teuthology.orchestra.run.smithi194.stdout:Scheduled mgr update... 2024-03-20T08:42:00.638 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:42:00 smithi194 ceph-mon[27237]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-20T08:42:00.638 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:42:00 smithi194 ceph-mon[27237]: Health check cleared: CEPHADM_STRAY_DAEMON (was: 1 stray daemon(s) not managed by cephadm) 2024-03-20T08:42:00.638 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:42:00 smithi194 ceph-mon[27237]: Cluster is now healthy 2024-03-20T08:42:00.704 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:42:00 smithi183 ceph-mon[25285]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-20T08:42:00.704 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:42:00 smithi183 ceph-mon[25285]: Health check cleared: CEPHADM_STRAY_DAEMON (was: 1 stray daemon(s) not managed by cephadm) 2024-03-20T08:42:00.704 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:42:00 smithi183 ceph-mon[25285]: Cluster is now healthy 2024-03-20T08:42:00.704 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:42:00 smithi183 ceph-mon[31860]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-20T08:42:00.704 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:42:00 smithi183 ceph-mon[31860]: Health check cleared: CEPHADM_STRAY_DAEMON (was: 1 stray daemon(s) not managed by cephadm) 2024-03-20T08:42:00.705 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:42:00 smithi183 ceph-mon[31860]: Cluster is now healthy 2024-03-20T08:42:01.120 DEBUG:teuthology.orchestra.run.smithi194:mgr.x> sudo journalctl -f -n 0 -u ceph-4c3f3eba-e695-11ee-95c9-87774f69a715@mgr.x.service 2024-03-20T08:42:01.122 INFO:tasks.cephadm:Deploying OSDs... 2024-03-20T08:42:01.122 DEBUG:teuthology.orchestra.run.smithi183:> set -ex 2024-03-20T08:42:01.122 DEBUG:teuthology.orchestra.run.smithi183:> dd if=/scratch_devs of=/dev/stdout 2024-03-20T08:42:01.141 DEBUG:teuthology.misc:devs=['/dev/nvme1n1', '/dev/nvme2n1', '/dev/nvme3n1', '/dev/nvme4n1'] 2024-03-20T08:42:01.142 DEBUG:teuthology.orchestra.run.smithi183:> stat /dev/nvme1n1 2024-03-20T08:42:01.199 INFO:teuthology.orchestra.run.smithi183.stdout: File: /dev/nvme1n1 2024-03-20T08:42:01.200 INFO:teuthology.orchestra.run.smithi183.stdout: Size: 0 Blocks: 0 IO Block: 4096 block special file 2024-03-20T08:42:01.200 INFO:teuthology.orchestra.run.smithi183.stdout:Device: 5h/5d Inode: 762 Links: 1 Device type: 103,2 2024-03-20T08:42:01.200 INFO:teuthology.orchestra.run.smithi183.stdout:Access: (0660/brw-rw----) Uid: ( 0/ root) Gid: ( 6/ disk) 2024-03-20T08:42:01.200 INFO:teuthology.orchestra.run.smithi183.stdout:Context: system_u:object_r:fixed_disk_device_t:s0 2024-03-20T08:42:01.200 INFO:teuthology.orchestra.run.smithi183.stdout:Access: 2024-03-20 08:41:54.693473630 +0000 2024-03-20T08:42:01.200 INFO:teuthology.orchestra.run.smithi183.stdout:Modify: 2024-03-20 08:38:57.532530140 +0000 2024-03-20T08:42:01.200 INFO:teuthology.orchestra.run.smithi183.stdout:Change: 2024-03-20 08:38:57.532530140 +0000 2024-03-20T08:42:01.200 INFO:teuthology.orchestra.run.smithi183.stdout: Birth: 2024-03-20 08:38:57.521530443 +0000 2024-03-20T08:42:01.200 DEBUG:teuthology.orchestra.run.smithi183:> sudo dd if=/dev/nvme1n1 of=/dev/null count=1 2024-03-20T08:42:01.269 INFO:teuthology.orchestra.run.smithi183.stderr:1+0 records in 2024-03-20T08:42:01.269 INFO:teuthology.orchestra.run.smithi183.stderr:1+0 records out 2024-03-20T08:42:01.269 INFO:teuthology.orchestra.run.smithi183.stderr:512 bytes copied, 0.000255268 s, 2.0 MB/s 2024-03-20T08:42:01.270 DEBUG:teuthology.orchestra.run.smithi183:> ! mount | grep -v devtmpfs | grep -q /dev/nvme1n1 2024-03-20T08:42:01.328 DEBUG:teuthology.orchestra.run.smithi183:> stat /dev/nvme2n1 2024-03-20T08:42:01.385 INFO:teuthology.orchestra.run.smithi183.stdout: File: /dev/nvme2n1 2024-03-20T08:42:01.385 INFO:teuthology.orchestra.run.smithi183.stdout: Size: 0 Blocks: 0 IO Block: 4096 block special file 2024-03-20T08:42:01.385 INFO:teuthology.orchestra.run.smithi183.stdout:Device: 5h/5d Inode: 775 Links: 1 Device type: 103,4 2024-03-20T08:42:01.385 INFO:teuthology.orchestra.run.smithi183.stdout:Access: (0660/brw-rw----) Uid: ( 0/ root) Gid: ( 6/ disk) 2024-03-20T08:42:01.385 INFO:teuthology.orchestra.run.smithi183.stdout:Context: system_u:object_r:fixed_disk_device_t:s0 2024-03-20T08:42:01.386 INFO:teuthology.orchestra.run.smithi183.stdout:Access: 2024-03-20 08:41:54.705473305 +0000 2024-03-20T08:42:01.386 INFO:teuthology.orchestra.run.smithi183.stdout:Modify: 2024-03-20 08:38:57.676526180 +0000 2024-03-20T08:42:01.386 INFO:teuthology.orchestra.run.smithi183.stdout:Change: 2024-03-20 08:38:57.676526180 +0000 2024-03-20T08:42:01.386 INFO:teuthology.orchestra.run.smithi183.stdout: Birth: 2024-03-20 08:38:57.664526510 +0000 2024-03-20T08:42:01.386 DEBUG:teuthology.orchestra.run.smithi183:> sudo dd if=/dev/nvme2n1 of=/dev/null count=1 2024-03-20T08:42:01.453 INFO:teuthology.orchestra.run.smithi183.stderr:1+0 records in 2024-03-20T08:42:01.453 INFO:teuthology.orchestra.run.smithi183.stderr:1+0 records out 2024-03-20T08:42:01.453 INFO:teuthology.orchestra.run.smithi183.stderr:512 bytes copied, 0.000218128 s, 2.3 MB/s 2024-03-20T08:42:01.454 DEBUG:teuthology.orchestra.run.smithi183:> ! mount | grep -v devtmpfs | grep -q /dev/nvme2n1 2024-03-20T08:42:01.511 DEBUG:teuthology.orchestra.run.smithi183:> stat /dev/nvme3n1 2024-03-20T08:42:01.568 INFO:teuthology.orchestra.run.smithi183.stdout: File: /dev/nvme3n1 2024-03-20T08:42:01.568 INFO:teuthology.orchestra.run.smithi183.stdout: Size: 0 Blocks: 0 IO Block: 4096 block special file 2024-03-20T08:42:01.568 INFO:teuthology.orchestra.run.smithi183.stdout:Device: 5h/5d Inode: 787 Links: 1 Device type: 103,6 2024-03-20T08:42:01.568 INFO:teuthology.orchestra.run.smithi183.stdout:Access: (0660/brw-rw----) Uid: ( 0/ root) Gid: ( 6/ disk) 2024-03-20T08:42:01.568 INFO:teuthology.orchestra.run.smithi183.stdout:Context: system_u:object_r:fixed_disk_device_t:s0 2024-03-20T08:42:01.568 INFO:teuthology.orchestra.run.smithi183.stdout:Access: 2024-03-20 08:41:54.717472980 +0000 2024-03-20T08:42:01.568 INFO:teuthology.orchestra.run.smithi183.stdout:Modify: 2024-03-20 08:38:57.819522247 +0000 2024-03-20T08:42:01.568 INFO:teuthology.orchestra.run.smithi183.stdout:Change: 2024-03-20 08:38:57.819522247 +0000 2024-03-20T08:42:01.568 INFO:teuthology.orchestra.run.smithi183.stdout: Birth: 2024-03-20 08:38:57.808522550 +0000 2024-03-20T08:42:01.569 DEBUG:teuthology.orchestra.run.smithi183:> sudo dd if=/dev/nvme3n1 of=/dev/null count=1 2024-03-20T08:42:01.624 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:42:01 smithi194 ceph-mon[27237]: from='client.24109 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "2;smithi183=y;smithi194=x", "target": ["mon-mgr", ""]}]: dispatch 2024-03-20T08:42:01.624 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:42:01 smithi194 ceph-mon[27237]: Saving service mgr spec with placement smithi183=y;smithi194=x;count:2 2024-03-20T08:42:01.624 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:42:01 smithi194 ceph-mon[27237]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:42:01.624 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:42:01 smithi194 ceph-mon[27237]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-20T08:42:01.624 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:42:01 smithi194 ceph-mon[27237]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-20T08:42:01.625 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:42:01 smithi194 ceph-mon[27237]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-20T08:42:01.625 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:42:01 smithi194 ceph-mon[27237]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:42:01.625 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:42:01 smithi194 ceph-mon[27237]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-20T08:42:01.625 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:42:01 smithi194 ceph-mon[27237]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-03-20T08:42:01.625 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:42:01 smithi194 ceph-mon[27237]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-20T08:42:01.625 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:42:01 smithi194 ceph-mon[27237]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-20T08:42:01.625 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:42:01 smithi194 ceph-mon[27237]: Deploying daemon mgr.x on smithi194 2024-03-20T08:42:01.633 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:42:01 smithi183 ceph-mon[25285]: from='client.24109 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "2;smithi183=y;smithi194=x", "target": ["mon-mgr", ""]}]: dispatch 2024-03-20T08:42:01.633 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:42:01 smithi183 ceph-mon[25285]: Saving service mgr spec with placement smithi183=y;smithi194=x;count:2 2024-03-20T08:42:01.633 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:42:01 smithi183 ceph-mon[25285]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:42:01.633 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:42:01 smithi183 ceph-mon[25285]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-20T08:42:01.633 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:42:01 smithi183 ceph-mon[25285]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-20T08:42:01.633 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:42:01 smithi183 ceph-mon[25285]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-20T08:42:01.633 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:42:01 smithi183 ceph-mon[25285]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:42:01.633 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:42:01 smithi183 ceph-mon[25285]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-20T08:42:01.633 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:42:01 smithi183 ceph-mon[25285]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-03-20T08:42:01.633 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:42:01 smithi183 ceph-mon[25285]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-20T08:42:01.633 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:42:01 smithi183 ceph-mon[25285]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-20T08:42:01.634 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:42:01 smithi183 ceph-mon[25285]: Deploying daemon mgr.x on smithi194 2024-03-20T08:42:01.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:42:01 smithi183 ceph-mon[31860]: from='client.24109 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "2;smithi183=y;smithi194=x", "target": ["mon-mgr", ""]}]: dispatch 2024-03-20T08:42:01.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:42:01 smithi183 ceph-mon[31860]: Saving service mgr spec with placement smithi183=y;smithi194=x;count:2 2024-03-20T08:42:01.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:42:01 smithi183 ceph-mon[31860]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:42:01.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:42:01 smithi183 ceph-mon[31860]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-20T08:42:01.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:42:01 smithi183 ceph-mon[31860]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-20T08:42:01.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:42:01 smithi183 ceph-mon[31860]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-20T08:42:01.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:42:01 smithi183 ceph-mon[31860]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:42:01.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:42:01 smithi183 ceph-mon[31860]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-20T08:42:01.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:42:01 smithi183 ceph-mon[31860]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-03-20T08:42:01.634 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:42:01 smithi183 ceph-mon[31860]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-20T08:42:01.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:42:01 smithi183 ceph-mon[31860]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-20T08:42:01.635 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:42:01 smithi183 ceph-mon[31860]: Deploying daemon mgr.x on smithi194 2024-03-20T08:42:01.635 INFO:teuthology.orchestra.run.smithi183.stderr:1+0 records in 2024-03-20T08:42:01.635 INFO:teuthology.orchestra.run.smithi183.stderr:1+0 records out 2024-03-20T08:42:01.635 INFO:teuthology.orchestra.run.smithi183.stderr:512 bytes copied, 0.000217862 s, 2.4 MB/s 2024-03-20T08:42:01.636 DEBUG:teuthology.orchestra.run.smithi183:> ! mount | grep -v devtmpfs | grep -q /dev/nvme3n1 2024-03-20T08:42:01.693 DEBUG:teuthology.orchestra.run.smithi183:> stat /dev/nvme4n1 2024-03-20T08:42:01.748 INFO:teuthology.orchestra.run.smithi183.stdout: File: /dev/nvme4n1 2024-03-20T08:42:01.748 INFO:teuthology.orchestra.run.smithi183.stdout: Size: 0 Blocks: 0 IO Block: 4096 block special file 2024-03-20T08:42:01.748 INFO:teuthology.orchestra.run.smithi183.stdout:Device: 5h/5d Inode: 799 Links: 1 Device type: 103,8 2024-03-20T08:42:01.748 INFO:teuthology.orchestra.run.smithi183.stdout:Access: (0660/brw-rw----) Uid: ( 0/ root) Gid: ( 6/ disk) 2024-03-20T08:42:01.748 INFO:teuthology.orchestra.run.smithi183.stdout:Context: system_u:object_r:fixed_disk_device_t:s0 2024-03-20T08:42:01.748 INFO:teuthology.orchestra.run.smithi183.stdout:Access: 2024-03-20 08:41:54.729472655 +0000 2024-03-20T08:42:01.748 INFO:teuthology.orchestra.run.smithi183.stdout:Modify: 2024-03-20 08:38:57.968518149 +0000 2024-03-20T08:42:01.748 INFO:teuthology.orchestra.run.smithi183.stdout:Change: 2024-03-20 08:38:57.968518149 +0000 2024-03-20T08:42:01.748 INFO:teuthology.orchestra.run.smithi183.stdout: Birth: 2024-03-20 08:38:57.957518452 +0000 2024-03-20T08:42:01.749 DEBUG:teuthology.orchestra.run.smithi183:> sudo dd if=/dev/nvme4n1 of=/dev/null count=1 2024-03-20T08:42:01.813 INFO:teuthology.orchestra.run.smithi183.stderr:1+0 records in 2024-03-20T08:42:01.814 INFO:teuthology.orchestra.run.smithi183.stderr:1+0 records out 2024-03-20T08:42:01.814 INFO:teuthology.orchestra.run.smithi183.stderr:512 bytes copied, 0.00023678 s, 2.2 MB/s 2024-03-20T08:42:01.815 DEBUG:teuthology.orchestra.run.smithi183:> ! mount | grep -v devtmpfs | grep -q /dev/nvme4n1 2024-03-20T08:42:01.872 DEBUG:teuthology.orchestra.run.smithi194:> set -ex 2024-03-20T08:42:01.872 DEBUG:teuthology.orchestra.run.smithi194:> dd if=/scratch_devs of=/dev/stdout 2024-03-20T08:42:01.889 DEBUG:teuthology.misc:devs=['/dev/nvme1n1', '/dev/nvme2n1', '/dev/nvme3n1', '/dev/nvme4n1'] 2024-03-20T08:42:01.889 DEBUG:teuthology.orchestra.run.smithi194:> stat /dev/nvme1n1 2024-03-20T08:42:01.945 INFO:teuthology.orchestra.run.smithi194.stdout: File: /dev/nvme1n1 2024-03-20T08:42:01.945 INFO:teuthology.orchestra.run.smithi194.stdout: Size: 0 Blocks: 0 IO Block: 4096 block special file 2024-03-20T08:42:01.945 INFO:teuthology.orchestra.run.smithi194.stdout:Device: 5h/5d Inode: 764 Links: 1 Device type: 103,2 2024-03-20T08:42:01.945 INFO:teuthology.orchestra.run.smithi194.stdout:Access: (0660/brw-rw----) Uid: ( 0/ root) Gid: ( 6/ disk) 2024-03-20T08:42:01.945 INFO:teuthology.orchestra.run.smithi194.stdout:Context: system_u:object_r:fixed_disk_device_t:s0 2024-03-20T08:42:01.945 INFO:teuthology.orchestra.run.smithi194.stdout:Access: 2024-03-20 08:41:55.246347680 +0000 2024-03-20T08:42:01.945 INFO:teuthology.orchestra.run.smithi194.stdout:Modify: 2024-03-20 08:38:59.168257040 +0000 2024-03-20T08:42:01.945 INFO:teuthology.orchestra.run.smithi194.stdout:Change: 2024-03-20 08:38:59.168257040 +0000 2024-03-20T08:42:01.945 INFO:teuthology.orchestra.run.smithi194.stdout: Birth: 2024-03-20 08:38:59.160257213 +0000 2024-03-20T08:42:01.946 DEBUG:teuthology.orchestra.run.smithi194:> sudo dd if=/dev/nvme1n1 of=/dev/null count=1 2024-03-20T08:42:02.012 INFO:teuthology.orchestra.run.smithi194.stderr:1+0 records in 2024-03-20T08:42:02.012 INFO:teuthology.orchestra.run.smithi194.stderr:1+0 records out 2024-03-20T08:42:02.012 INFO:teuthology.orchestra.run.smithi194.stderr:512 bytes copied, 0.000258385 s, 2.0 MB/s 2024-03-20T08:42:02.013 DEBUG:teuthology.orchestra.run.smithi194:> ! mount | grep -v devtmpfs | grep -q /dev/nvme1n1 2024-03-20T08:42:02.070 DEBUG:teuthology.orchestra.run.smithi194:> stat /dev/nvme2n1 2024-03-20T08:42:02.125 INFO:teuthology.orchestra.run.smithi194.stdout: File: /dev/nvme2n1 2024-03-20T08:42:02.126 INFO:teuthology.orchestra.run.smithi194.stdout: Size: 0 Blocks: 0 IO Block: 4096 block special file 2024-03-20T08:42:02.126 INFO:teuthology.orchestra.run.smithi194.stdout:Device: 5h/5d Inode: 774 Links: 1 Device type: 103,4 2024-03-20T08:42:02.126 INFO:teuthology.orchestra.run.smithi194.stdout:Access: (0660/brw-rw----) Uid: ( 0/ root) Gid: ( 6/ disk) 2024-03-20T08:42:02.126 INFO:teuthology.orchestra.run.smithi194.stdout:Context: system_u:object_r:fixed_disk_device_t:s0 2024-03-20T08:42:02.126 INFO:teuthology.orchestra.run.smithi194.stdout:Access: 2024-03-20 08:41:55.257347440 +0000 2024-03-20T08:42:02.126 INFO:teuthology.orchestra.run.smithi194.stdout:Modify: 2024-03-20 08:38:59.316253854 +0000 2024-03-20T08:42:02.126 INFO:teuthology.orchestra.run.smithi194.stdout:Change: 2024-03-20 08:38:59.316253854 +0000 2024-03-20T08:42:02.126 INFO:teuthology.orchestra.run.smithi194.stdout: Birth: 2024-03-20 08:38:59.305254091 +0000 2024-03-20T08:42:02.126 DEBUG:teuthology.orchestra.run.smithi194:> sudo dd if=/dev/nvme2n1 of=/dev/null count=1 2024-03-20T08:42:02.192 INFO:teuthology.orchestra.run.smithi194.stderr:1+0 records in 2024-03-20T08:42:02.192 INFO:teuthology.orchestra.run.smithi194.stderr:1+0 records out 2024-03-20T08:42:02.192 INFO:teuthology.orchestra.run.smithi194.stderr:512 bytes copied, 0.000219601 s, 2.3 MB/s 2024-03-20T08:42:02.194 DEBUG:teuthology.orchestra.run.smithi194:> ! mount | grep -v devtmpfs | grep -q /dev/nvme2n1 2024-03-20T08:42:02.249 DEBUG:teuthology.orchestra.run.smithi194:> stat /dev/nvme3n1 2024-03-20T08:42:02.307 INFO:teuthology.orchestra.run.smithi194.stdout: File: /dev/nvme3n1 2024-03-20T08:42:02.307 INFO:teuthology.orchestra.run.smithi194.stdout: Size: 0 Blocks: 0 IO Block: 4096 block special file 2024-03-20T08:42:02.307 INFO:teuthology.orchestra.run.smithi194.stdout:Device: 5h/5d Inode: 787 Links: 1 Device type: 103,6 2024-03-20T08:42:02.307 INFO:teuthology.orchestra.run.smithi194.stdout:Access: (0660/brw-rw----) Uid: ( 0/ root) Gid: ( 6/ disk) 2024-03-20T08:42:02.307 INFO:teuthology.orchestra.run.smithi194.stdout:Context: system_u:object_r:fixed_disk_device_t:s0 2024-03-20T08:42:02.307 INFO:teuthology.orchestra.run.smithi194.stdout:Access: 2024-03-20 08:41:55.268347199 +0000 2024-03-20T08:42:02.308 INFO:teuthology.orchestra.run.smithi194.stdout:Modify: 2024-03-20 08:38:59.465250645 +0000 2024-03-20T08:42:02.308 INFO:teuthology.orchestra.run.smithi194.stdout:Change: 2024-03-20 08:38:59.465250645 +0000 2024-03-20T08:42:02.308 INFO:teuthology.orchestra.run.smithi194.stdout: Birth: 2024-03-20 08:38:59.448251012 +0000 2024-03-20T08:42:02.308 DEBUG:teuthology.orchestra.run.smithi194:> sudo dd if=/dev/nvme3n1 of=/dev/null count=1 2024-03-20T08:42:02.525 INFO:teuthology.orchestra.run.smithi194.stderr:1+0 records in 2024-03-20T08:42:02.525 INFO:teuthology.orchestra.run.smithi194.stderr:1+0 records out 2024-03-20T08:42:02.525 INFO:teuthology.orchestra.run.smithi194.stderr:512 bytes copied, 0.000192949 s, 2.7 MB/s 2024-03-20T08:42:02.527 DEBUG:teuthology.orchestra.run.smithi194:> ! mount | grep -v devtmpfs | grep -q /dev/nvme3n1 2024-03-20T08:42:02.542 DEBUG:teuthology.orchestra.run.smithi194:> stat /dev/nvme4n1 2024-03-20T08:42:02.602 INFO:teuthology.orchestra.run.smithi194.stdout: File: /dev/nvme4n1 2024-03-20T08:42:02.602 INFO:teuthology.orchestra.run.smithi194.stdout: Size: 0 Blocks: 0 IO Block: 4096 block special file 2024-03-20T08:42:02.602 INFO:teuthology.orchestra.run.smithi194.stdout:Device: 5h/5d Inode: 799 Links: 1 Device type: 103,8 2024-03-20T08:42:02.602 INFO:teuthology.orchestra.run.smithi194.stdout:Access: (0660/brw-rw----) Uid: ( 0/ root) Gid: ( 6/ disk) 2024-03-20T08:42:02.602 INFO:teuthology.orchestra.run.smithi194.stdout:Context: system_u:object_r:fixed_disk_device_t:s0 2024-03-20T08:42:02.602 INFO:teuthology.orchestra.run.smithi194.stdout:Access: 2024-03-20 08:41:55.278346981 +0000 2024-03-20T08:42:02.602 INFO:teuthology.orchestra.run.smithi194.stdout:Modify: 2024-03-20 08:38:59.604247652 +0000 2024-03-20T08:42:02.603 INFO:teuthology.orchestra.run.smithi194.stdout:Change: 2024-03-20 08:38:59.604247652 +0000 2024-03-20T08:42:02.603 INFO:teuthology.orchestra.run.smithi194.stdout: Birth: 2024-03-20 08:38:59.592247911 +0000 2024-03-20T08:42:02.603 DEBUG:teuthology.orchestra.run.smithi194:> sudo dd if=/dev/nvme4n1 of=/dev/null count=1 2024-03-20T08:42:02.715 INFO:teuthology.orchestra.run.smithi194.stderr:1+0 records in 2024-03-20T08:42:02.716 INFO:teuthology.orchestra.run.smithi194.stderr:1+0 records out 2024-03-20T08:42:02.716 INFO:teuthology.orchestra.run.smithi194.stderr:512 bytes copied, 0.000199626 s, 2.6 MB/s 2024-03-20T08:42:02.717 DEBUG:teuthology.orchestra.run.smithi194:> ! mount | grep -v devtmpfs | grep -q /dev/nvme4n1 2024-03-20T08:42:02.731 INFO:tasks.cephadm:Deploying osd.0 on smithi183 with /dev/nvme4n1... 2024-03-20T08:42:02.731 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4c3f3eba-e695-11ee-95c9-87774f69a715 -- ceph-volume lvm zap /dev/nvme4n1 2024-03-20T08:42:02.888 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:42:02 smithi194 ceph-mon[27237]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-20T08:42:02.889 INFO:journalctl@ceph.mgr.x.smithi194.stdout:Mar 20 08:42:02 smithi194 systemd[1]: Starting Ceph mgr.x for 4c3f3eba-e695-11ee-95c9-87774f69a715... 2024-03-20T08:42:02.954 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:42:02 smithi183 ceph-mon[25285]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-20T08:42:02.954 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:42:02 smithi183 ceph-mon[31860]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-20T08:42:02.977 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/4c3f3eba-e695-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T08:42:03.256 INFO:journalctl@ceph.mgr.x.smithi194.stdout:Mar 20 08:42:02 smithi194 podman[29657]: 2024-03-20 08:42:02.85195334 +0000 UTC m=+0.017181937 image pull 9ceba81fb2c8d8682cb159527017d591637b153b3aa4a390ffa8586db6dcda00 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:976a8b2d06e3c6bfd12488297e8584dd704cc06c3e976c7e0f718c13e238e912 2024-03-20T08:42:03.257 INFO:journalctl@ceph.mgr.x.smithi194.stdout:Mar 20 08:42:02 smithi194 podman[29657]: 2024-03-20 08:42:02.971524296 +0000 UTC m=+0.136752885 container create 625764fcda26def3fba993831c484f9fe43d54b4951ee495cd88855b47232a6e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:976a8b2d06e3c6bfd12488297e8584dd704cc06c3e976c7e0f718c13e238e912, name=ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-mgr-x, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.build-date=20240307, org.label-schema.schema-version=1.0, ceph=True, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-yuri5-testing-2024-03-18-1144-e7e80a6, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-20T08:42:03.257 INFO:journalctl@ceph.mgr.x.smithi194.stdout:Mar 20 08:42:03 smithi194 podman[29657]: 2024-03-20 08:42:03.165400488 +0000 UTC m=+0.330629078 container init 625764fcda26def3fba993831c484f9fe43d54b4951ee495cd88855b47232a6e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:976a8b2d06e3c6bfd12488297e8584dd704cc06c3e976c7e0f718c13e238e912, name=ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-mgr-x, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, org.label-schema.build-date=20240307, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, ceph=True, RELEASE=wip-yuri5-testing-2024-03-18-1144-e7e80a6, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD) 2024-03-20T08:42:03.257 INFO:journalctl@ceph.mgr.x.smithi194.stdout:Mar 20 08:42:03 smithi194 podman[29657]: 2024-03-20 08:42:03.168556606 +0000 UTC m=+0.333785189 container start 625764fcda26def3fba993831c484f9fe43d54b4951ee495cd88855b47232a6e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:976a8b2d06e3c6bfd12488297e8584dd704cc06c3e976c7e0f718c13e238e912, name=ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-mgr-x, org.label-schema.build-date=20240307, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_CLEAN=True, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-yuri5-testing-2024-03-18-1144-e7e80a6, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2) 2024-03-20T08:42:03.257 INFO:journalctl@ceph.mgr.x.smithi194.stdout:Mar 20 08:42:03 smithi194 ceph-mgr[29676]: -- 172.21.15.194:0/1954080397 <== mon.1 v2:172.21.15.194:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55e6d6da25a0 con 0x55e6d6d77000 2024-03-20T08:42:03.257 INFO:journalctl@ceph.mgr.x.smithi194.stdout:Mar 20 08:42:03 smithi194 bash[29657]: 625764fcda26def3fba993831c484f9fe43d54b4951ee495cd88855b47232a6e 2024-03-20T08:42:03.638 INFO:journalctl@ceph.mgr.x.smithi194.stdout:Mar 20 08:42:03 smithi194 systemd[1]: Started Ceph mgr.x for 4c3f3eba-e695-11ee-95c9-87774f69a715. 2024-03-20T08:42:04.042 INFO:journalctl@ceph.mgr.x.smithi194.stdout:Mar 20 08:42:03 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-mgr-x[29672]: 2024-03-20T08:42:03.732+0000 7f1419a34200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-20T08:42:04.043 INFO:journalctl@ceph.mgr.x.smithi194.stdout:Mar 20 08:42:03 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-mgr-x[29672]: 2024-03-20T08:42:03.826+0000 7f1419a34200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-20T08:42:04.589 INFO:journalctl@ceph.mgr.x.smithi194.stdout:Mar 20 08:42:04 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-mgr-x[29672]: 2024-03-20T08:42:04.496+0000 7f1419a34200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-20T08:42:04.589 INFO:journalctl@ceph.mgr.x.smithi194.stdout:Mar 20 08:42:04 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-mgr-x[29672]: 2024-03-20T08:42:04.589+0000 7f1419a34200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-20T08:42:04.590 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:42:04 smithi194 ceph-mon[27237]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-20T08:42:04.590 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:42:04 smithi194 ceph-mon[27237]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:42:04.590 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:42:04 smithi194 ceph-mon[27237]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:42:04.590 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:42:04 smithi194 ceph-mon[27237]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:42:04.590 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:42:04 smithi194 ceph-mon[27237]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:42:04.590 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:42:04 smithi194 ceph-mon[27237]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-20T08:42:04.590 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:42:04 smithi194 ceph-mon[27237]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-20T08:42:04.590 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:42:04 smithi194 ceph-mon[27237]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-20T08:42:04.590 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:42:04 smithi194 ceph-mon[27237]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:42:04.590 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:42:04 smithi194 ceph-mon[27237]: Metadata not up to date on all hosts. Skipping non agent specs 2024-03-20T08:42:04.591 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:42:04 smithi194 ceph-mon[27237]: Reconfiguring mgr.y (unknown last config time)... 2024-03-20T08:42:04.591 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:42:04 smithi194 ceph-mon[27237]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-20T08:42:04.591 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:42:04 smithi194 ceph-mon[27237]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-20T08:42:04.591 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:42:04 smithi194 ceph-mon[27237]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-20T08:42:04.591 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:42:04 smithi194 ceph-mon[27237]: Reconfiguring daemon mgr.y on smithi183 2024-03-20T08:42:04.704 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:42:04 smithi183 ceph-mon[25285]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-20T08:42:04.704 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:42:04 smithi183 ceph-mon[25285]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:42:04.704 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:42:04 smithi183 ceph-mon[25285]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:42:04.705 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:42:04 smithi183 ceph-mon[25285]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:42:04.705 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:42:04 smithi183 ceph-mon[25285]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:42:04.705 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:42:04 smithi183 ceph-mon[25285]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-20T08:42:04.705 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:42:04 smithi183 ceph-mon[25285]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-20T08:42:04.705 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:42:04 smithi183 ceph-mon[25285]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-20T08:42:04.705 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:42:04 smithi183 ceph-mon[25285]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:42:04.705 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:42:04 smithi183 ceph-mon[25285]: Metadata not up to date on all hosts. Skipping non agent specs 2024-03-20T08:42:04.705 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:42:04 smithi183 ceph-mon[25285]: Reconfiguring mgr.y (unknown last config time)... 2024-03-20T08:42:04.705 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:42:04 smithi183 ceph-mon[25285]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-20T08:42:04.705 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:42:04 smithi183 ceph-mon[25285]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-20T08:42:04.705 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:42:04 smithi183 ceph-mon[25285]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-20T08:42:04.705 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:42:04 smithi183 ceph-mon[25285]: Reconfiguring daemon mgr.y on smithi183 2024-03-20T08:42:04.706 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:42:04 smithi183 ceph-mon[31860]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-20T08:42:04.706 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:42:04 smithi183 ceph-mon[31860]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:42:04.706 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:42:04 smithi183 ceph-mon[31860]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:42:04.706 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:42:04 smithi183 ceph-mon[31860]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:42:04.706 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:42:04 smithi183 ceph-mon[31860]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:42:04.706 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:42:04 smithi183 ceph-mon[31860]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-20T08:42:04.706 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:42:04 smithi183 ceph-mon[31860]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-20T08:42:04.706 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:42:04 smithi183 ceph-mon[31860]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-20T08:42:04.706 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:42:04 smithi183 ceph-mon[31860]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:42:04.706 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:42:04 smithi183 ceph-mon[31860]: Metadata not up to date on all hosts. Skipping non agent specs 2024-03-20T08:42:04.706 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:42:04 smithi183 ceph-mon[31860]: Reconfiguring mgr.y (unknown last config time)... 2024-03-20T08:42:04.706 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:42:04 smithi183 ceph-mon[31860]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-20T08:42:04.706 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:42:04 smithi183 ceph-mon[31860]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-20T08:42:04.706 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:42:04 smithi183 ceph-mon[31860]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-20T08:42:04.707 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:42:04 smithi183 ceph-mon[31860]: Reconfiguring daemon mgr.y on smithi183 2024-03-20T08:42:04.839 INFO:journalctl@ceph.mgr.x.smithi194.stdout:Mar 20 08:42:04 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-mgr-x[29672]: 2024-03-20T08:42:04.734+0000 7f1419a34200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-20T08:42:04.840 INFO:journalctl@ceph.mgr.x.smithi194.stdout:Mar 20 08:42:04 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-mgr-x[29672]: 2024-03-20T08:42:04.839+0000 7f1419a34200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-20T08:42:05.099 INFO:journalctl@ceph.mgr.x.smithi194.stdout:Mar 20 08:42:04 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-mgr-x[29672]: 2024-03-20T08:42:04.924+0000 7f1419a34200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-20T08:42:05.388 INFO:journalctl@ceph.mgr.x.smithi194.stdout:Mar 20 08:42:05 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-mgr-x[29672]: 2024-03-20T08:42:05.202+0000 7f1419a34200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-20T08:42:05.388 INFO:journalctl@ceph.mgr.x.smithi194.stdout:Mar 20 08:42:05 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-mgr-x[29672]: 2024-03-20T08:42:05.322+0000 7f1419a34200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-20T08:42:05.580 INFO:teuthology.orchestra.run.smithi183.stderr:--> Zapping: /dev/nvme4n1 2024-03-20T08:42:05.580 INFO:teuthology.orchestra.run.smithi183.stderr:--> --destroy was not specified, but zapping a whole device will remove the partition table 2024-03-20T08:42:05.581 INFO:teuthology.orchestra.run.smithi183.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/nvme4n1 bs=1M count=10 conv=fsync 2024-03-20T08:42:05.581 INFO:teuthology.orchestra.run.smithi183.stderr: stderr: 10+0 records in 2024-03-20T08:42:05.581 INFO:teuthology.orchestra.run.smithi183.stderr:10+0 records out 2024-03-20T08:42:05.581 INFO:teuthology.orchestra.run.smithi183.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0155745 s, 673 MB/s 2024-03-20T08:42:05.581 INFO:teuthology.orchestra.run.smithi183.stderr:--> Zapping successful for: 2024-03-20T08:42:05.703 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:42:05 smithi194 ceph-mon[27237]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:42:05.704 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:42:05 smithi194 ceph-mon[27237]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:42:05.704 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:42:05 smithi194 ceph-mon[27237]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:42:05.704 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:42:05 smithi194 ceph-mon[27237]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-20T08:42:05.704 INFO:journalctl@ceph.mgr.x.smithi194.stdout:Mar 20 08:42:05 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-mgr-x[29672]: 2024-03-20T08:42:05.425+0000 7f1419a34200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-20T08:42:05.704 INFO:journalctl@ceph.mgr.x.smithi194.stdout:Mar 20 08:42:05 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-mgr-x[29672]: 2024-03-20T08:42:05.544+0000 7f1419a34200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-20T08:42:05.705 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:42:05 smithi183 ceph-mon[25285]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:42:05.705 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:42:05 smithi183 ceph-mon[25285]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:42:05.705 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:42:05 smithi183 ceph-mon[25285]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:42:05.705 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:42:05 smithi183 ceph-mon[25285]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-20T08:42:05.705 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:42:05 smithi183 ceph-mon[31860]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:42:05.705 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:42:05 smithi183 ceph-mon[31860]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:42:05.705 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:42:05 smithi183 ceph-mon[31860]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:42:05.705 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:42:05 smithi183 ceph-mon[31860]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-20T08:42:06.138 INFO:journalctl@ceph.mgr.x.smithi194.stdout:Mar 20 08:42:05 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-mgr-x[29672]: 2024-03-20T08:42:05.735+0000 7f1419a34200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-20T08:42:06.248 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4c3f3eba-e695-11ee-95c9-87774f69a715 -- ceph orch daemon add osd smithi183:/dev/nvme4n1 2024-03-20T08:42:06.489 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/4c3f3eba-e695-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T08:42:06.864 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:42:06 smithi194 ceph-mon[27237]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-20T08:42:06.865 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:42:06 smithi194 ceph-mon[27237]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-20T08:42:06.865 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:42:06 smithi194 ceph-mon[27237]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-20T08:42:06.865 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:42:06 smithi194 ceph-mon[27237]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:42:06.865 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:42:06 smithi194 ceph-mon[27237]: Metadata not up to date on all hosts. Skipping non agent specs 2024-03-20T08:42:06.865 INFO:journalctl@ceph.mgr.x.smithi194.stdout:Mar 20 08:42:06 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-mgr-x[29672]: 2024-03-20T08:42:06.726+0000 7f1419a34200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-20T08:42:06.897 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:42:06 smithi183 ceph-mon[31860]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-20T08:42:06.897 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:42:06 smithi183 ceph-mon[31860]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-20T08:42:06.897 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:42:06 smithi183 ceph-mon[31860]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-20T08:42:06.897 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:42:06 smithi183 ceph-mon[31860]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:42:06.898 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:42:06 smithi183 ceph-mon[31860]: Metadata not up to date on all hosts. Skipping non agent specs 2024-03-20T08:42:06.898 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:42:06 smithi183 ceph-mon[25285]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-20T08:42:06.898 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:42:06 smithi183 ceph-mon[25285]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-20T08:42:06.898 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:42:06 smithi183 ceph-mon[25285]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-20T08:42:06.898 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:42:06 smithi183 ceph-mon[25285]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:42:06.898 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:42:06 smithi183 ceph-mon[25285]: Metadata not up to date on all hosts. Skipping non agent specs 2024-03-20T08:42:07.138 INFO:journalctl@ceph.mgr.x.smithi194.stdout:Mar 20 08:42:06 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-mgr-x[29672]: 2024-03-20T08:42:06.864+0000 7f1419a34200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-20T08:42:07.495 INFO:journalctl@ceph.mgr.x.smithi194.stdout:Mar 20 08:42:07 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-mgr-x[29672]: 2024-03-20T08:42:07.209+0000 7f1419a34200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-20T08:42:07.495 INFO:journalctl@ceph.mgr.x.smithi194.stdout:Mar 20 08:42:07 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-mgr-x[29672]: 2024-03-20T08:42:07.298+0000 7f1419a34200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-20T08:42:07.495 INFO:journalctl@ceph.mgr.x.smithi194.stdout:Mar 20 08:42:07 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-mgr-x[29672]: 2024-03-20T08:42:07.495+0000 7f1419a34200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-20T08:42:07.869 INFO:journalctl@ceph.mgr.x.smithi194.stdout:Mar 20 08:42:07 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-mgr-x[29672]: 2024-03-20T08:42:07.681+0000 7f1419a34200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-20T08:42:08.138 INFO:journalctl@ceph.mgr.x.smithi194.stdout:Mar 20 08:42:07 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-mgr-x[29672]: 2024-03-20T08:42:07.968+0000 7f1419a34200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-20T08:42:08.138 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:42:07 smithi194 ceph-mon[27237]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:42:08.139 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:42:07 smithi194 ceph-mon[27237]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-20T08:42:08.139 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:42:07 smithi194 ceph-mon[27237]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:42:08.139 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:42:07 smithi194 ceph-mon[27237]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:42:08.139 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:42:07 smithi194 ceph-mon[27237]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:42:08.139 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:42:07 smithi194 ceph-mon[27237]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-20T08:42:08.139 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:42:07 smithi194 ceph-mon[27237]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-20T08:42:08.139 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:42:07 smithi194 ceph-mon[27237]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-20T08:42:08.139 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:42:07 smithi194 ceph-mon[27237]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:42:08.204 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:42:07 smithi183 ceph-mon[25285]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:42:08.204 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:42:07 smithi183 ceph-mon[25285]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-20T08:42:08.205 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:42:07 smithi183 ceph-mon[25285]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:42:08.205 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:42:07 smithi183 ceph-mon[25285]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:42:08.205 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:42:07 smithi183 ceph-mon[25285]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:42:08.205 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:42:07 smithi183 ceph-mon[25285]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-20T08:42:08.205 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:42:07 smithi183 ceph-mon[25285]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-20T08:42:08.205 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:42:07 smithi183 ceph-mon[25285]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-20T08:42:08.205 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:42:07 smithi183 ceph-mon[25285]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:42:08.205 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:42:07 smithi183 ceph-mon[31860]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:42:08.205 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:42:07 smithi183 ceph-mon[31860]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-20T08:42:08.205 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:42:07 smithi183 ceph-mon[31860]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:42:08.205 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:42:07 smithi183 ceph-mon[31860]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:42:08.205 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:42:07 smithi183 ceph-mon[31860]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:42:08.206 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:42:07 smithi183 ceph-mon[31860]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-20T08:42:08.206 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:42:07 smithi183 ceph-mon[31860]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-20T08:42:08.206 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:42:07 smithi183 ceph-mon[31860]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-20T08:42:08.206 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:42:07 smithi183 ceph-mon[31860]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:42:08.638 INFO:journalctl@ceph.mgr.x.smithi194.stdout:Mar 20 08:42:08 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-mgr-x[29672]: 2024-03-20T08:42:08.231+0000 7f1419a34200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-20T08:42:08.638 INFO:journalctl@ceph.mgr.x.smithi194.stdout:Mar 20 08:42:08 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-mgr-x[29672]: 2024-03-20T08:42:08.466+0000 7f1419a34200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-20T08:42:09.139 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:42:08 smithi194 ceph-mon[27237]: from='client.14217 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi183:/dev/nvme4n1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-20T08:42:09.139 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:42:08 smithi194 ceph-mon[27237]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-20T08:42:09.139 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:42:08 smithi194 ceph-mon[27237]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-20T08:42:09.139 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:42:08 smithi194 ceph-mon[27237]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-20T08:42:09.178 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:42:08 smithi183 ceph-mon[25285]: from='client.14217 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi183:/dev/nvme4n1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-20T08:42:09.179 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:42:08 smithi183 ceph-mon[25285]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-20T08:42:09.179 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:42:08 smithi183 ceph-mon[25285]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-20T08:42:09.179 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:42:08 smithi183 ceph-mon[25285]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-20T08:42:09.179 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:42:08 smithi183 ceph-mon[31860]: from='client.14217 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi183:/dev/nvme4n1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-20T08:42:09.179 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:42:08 smithi183 ceph-mon[31860]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-20T08:42:09.179 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:42:08 smithi183 ceph-mon[31860]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-20T08:42:09.179 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:42:08 smithi183 ceph-mon[31860]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-20T08:42:09.638 INFO:journalctl@ceph.mgr.x.smithi194.stdout:Mar 20 08:42:09 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-mgr-x[29672]: 2024-03-20T08:42:09.257+0000 7f1419a34200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-20T08:42:09.880 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:42:09 smithi183 ceph-mon[25285]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-20T08:42:10.136 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:42:09 smithi183 ceph-mon[31860]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-20T08:42:10.138 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:42:09 smithi194 ceph-mon[27237]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-20T08:42:10.138 INFO:journalctl@ceph.mgr.x.smithi194.stdout:Mar 20 08:42:09 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-mgr-x[29672]: 2024-03-20T08:42:09.826+0000 7f1419a34200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-20T08:42:10.138 INFO:journalctl@ceph.mgr.x.smithi194.stdout:Mar 20 08:42:09 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-mgr-x[29672]: 2024-03-20T08:42:09.924+0000 7f1419a34200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-20T08:42:10.638 INFO:journalctl@ceph.mgr.x.smithi194.stdout:Mar 20 08:42:10 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-mgr-x[29672]: 2024-03-20T08:42:10.141+0000 7f1419a34200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-20T08:42:11.138 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:42:10 smithi194 ceph-mon[27237]: Standby manager daemon x started 2024-03-20T08:42:11.138 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:42:10 smithi194 ceph-mon[27237]: from='mgr.? 172.21.15.194:0/3084970885' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-03-20T08:42:11.139 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:42:10 smithi194 ceph-mon[27237]: from='mgr.? 172.21.15.194:0/3084970885' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-20T08:42:11.139 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:42:10 smithi194 ceph-mon[27237]: from='mgr.? 172.21.15.194:0/3084970885' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-03-20T08:42:11.139 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:42:10 smithi194 ceph-mon[27237]: from='mgr.? 172.21.15.194:0/3084970885' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-20T08:42:11.139 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:42:10 smithi194 ceph-mon[27237]: from='client.? 172.21.15.183:0/2885234284' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "8ed0dd19-c4b0-4768-b28c-957c19afa5b4"}]: dispatch 2024-03-20T08:42:11.139 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:42:10 smithi194 ceph-mon[27237]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "8ed0dd19-c4b0-4768-b28c-957c19afa5b4"}]: dispatch 2024-03-20T08:42:11.204 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:42:10 smithi183 ceph-mon[31860]: Standby manager daemon x started 2024-03-20T08:42:11.205 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:42:10 smithi183 ceph-mon[31860]: from='mgr.? 172.21.15.194:0/3084970885' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-03-20T08:42:11.205 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:42:10 smithi183 ceph-mon[31860]: from='mgr.? 172.21.15.194:0/3084970885' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-20T08:42:11.205 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:42:10 smithi183 ceph-mon[31860]: from='mgr.? 172.21.15.194:0/3084970885' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-03-20T08:42:11.205 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:42:10 smithi183 ceph-mon[31860]: from='mgr.? 172.21.15.194:0/3084970885' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-20T08:42:11.205 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:42:10 smithi183 ceph-mon[31860]: from='client.? 172.21.15.183:0/2885234284' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "8ed0dd19-c4b0-4768-b28c-957c19afa5b4"}]: dispatch 2024-03-20T08:42:11.205 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:42:10 smithi183 ceph-mon[31860]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "8ed0dd19-c4b0-4768-b28c-957c19afa5b4"}]: dispatch 2024-03-20T08:42:11.205 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:42:10 smithi183 ceph-mon[25285]: Standby manager daemon x started 2024-03-20T08:42:11.205 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:42:10 smithi183 ceph-mon[25285]: from='mgr.? 172.21.15.194:0/3084970885' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-03-20T08:42:11.205 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:42:10 smithi183 ceph-mon[25285]: from='mgr.? 172.21.15.194:0/3084970885' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-20T08:42:11.205 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:42:10 smithi183 ceph-mon[25285]: from='mgr.? 172.21.15.194:0/3084970885' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-03-20T08:42:11.205 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:42:10 smithi183 ceph-mon[25285]: from='mgr.? 172.21.15.194:0/3084970885' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-20T08:42:11.205 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:42:10 smithi183 ceph-mon[25285]: from='client.? 172.21.15.183:0/2885234284' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "8ed0dd19-c4b0-4768-b28c-957c19afa5b4"}]: dispatch 2024-03-20T08:42:11.206 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:42:10 smithi183 ceph-mon[25285]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "8ed0dd19-c4b0-4768-b28c-957c19afa5b4"}]: dispatch 2024-03-20T08:42:12.204 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:42:11 smithi183 ceph-mon[25285]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "8ed0dd19-c4b0-4768-b28c-957c19afa5b4"}]': finished 2024-03-20T08:42:12.204 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:42:11 smithi183 ceph-mon[25285]: osdmap e5: 1 total, 0 up, 1 in 2024-03-20T08:42:12.204 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:42:11 smithi183 ceph-mon[25285]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-20T08:42:12.205 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:42:11 smithi183 ceph-mon[25285]: mgrmap e14: y(active, since 71s), standbys: x 2024-03-20T08:42:12.205 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:42:11 smithi183 ceph-mon[25285]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-20T08:42:12.205 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:42:11 smithi183 ceph-mon[25285]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-20T08:42:12.205 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:42:11 smithi183 ceph-mon[31860]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "8ed0dd19-c4b0-4768-b28c-957c19afa5b4"}]': finished 2024-03-20T08:42:12.205 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:42:11 smithi183 ceph-mon[31860]: osdmap e5: 1 total, 0 up, 1 in 2024-03-20T08:42:12.205 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:42:11 smithi183 ceph-mon[31860]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-20T08:42:12.205 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:42:11 smithi183 ceph-mon[31860]: mgrmap e14: y(active, since 71s), standbys: x 2024-03-20T08:42:12.205 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:42:11 smithi183 ceph-mon[31860]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-20T08:42:12.205 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:42:11 smithi183 ceph-mon[31860]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-20T08:42:12.388 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:42:11 smithi194 ceph-mon[27237]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "8ed0dd19-c4b0-4768-b28c-957c19afa5b4"}]': finished 2024-03-20T08:42:12.389 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:42:11 smithi194 ceph-mon[27237]: osdmap e5: 1 total, 0 up, 1 in 2024-03-20T08:42:12.389 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:42:11 smithi194 ceph-mon[27237]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-20T08:42:12.389 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:42:11 smithi194 ceph-mon[27237]: mgrmap e14: y(active, since 71s), standbys: x 2024-03-20T08:42:12.389 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:42:11 smithi194 ceph-mon[27237]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-20T08:42:12.389 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:42:11 smithi194 ceph-mon[27237]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-20T08:42:13.150 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:42:12 smithi183 ceph-mon[25285]: from='client.? 172.21.15.183:0/4288387185' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-20T08:42:13.150 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:42:12 smithi183 ceph-mon[31860]: from='client.? 172.21.15.183:0/4288387185' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-20T08:42:13.388 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:42:12 smithi194 ceph-mon[27237]: from='client.? 172.21.15.183:0/4288387185' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-20T08:42:14.168 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:42:13 smithi183 ceph-mon[25285]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-20T08:42:14.168 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:42:13 smithi183 ceph-mon[31860]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-20T08:42:14.388 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:42:13 smithi194 ceph-mon[27237]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-20T08:42:16.204 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:42:15 smithi183 ceph-mon[25285]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-20T08:42:16.205 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:42:15 smithi183 ceph-mon[31860]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-20T08:42:16.388 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:42:15 smithi194 ceph-mon[27237]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-20T08:42:17.454 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:42:17 smithi183 ceph-mon[25285]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:42:17.454 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:42:17 smithi183 ceph-mon[25285]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:42:17.455 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:42:17 smithi183 ceph-mon[25285]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:42:17.455 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:42:17 smithi183 ceph-mon[31860]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:42:17.455 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:42:17 smithi183 ceph-mon[31860]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:42:17.455 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:42:17 smithi183 ceph-mon[31860]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:42:17.638 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:42:17 smithi194 ceph-mon[27237]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:42:17.638 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:42:17 smithi194 ceph-mon[27237]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:42:17.638 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:42:17 smithi194 ceph-mon[27237]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:42:18.559 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:42:18 smithi183 ceph-mon[25285]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-20T08:42:18.559 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:42:18 smithi183 ceph-mon[31860]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-20T08:42:18.638 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:42:18 smithi194 ceph-mon[27237]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-20T08:42:19.454 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:42:19 smithi183 ceph-mon[31860]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-03-20T08:42:19.455 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:42:19 smithi183 ceph-mon[31860]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-20T08:42:19.455 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:42:19 smithi183 ceph-mon[31860]: Deploying daemon osd.0 on smithi183 2024-03-20T08:42:19.455 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:42:19 smithi183 ceph-mon[25285]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-03-20T08:42:19.456 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:42:19 smithi183 ceph-mon[25285]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-20T08:42:19.456 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:42:19 smithi183 ceph-mon[25285]: Deploying daemon osd.0 on smithi183 2024-03-20T08:42:19.638 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:42:19 smithi194 ceph-mon[27237]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-03-20T08:42:19.639 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:42:19 smithi194 ceph-mon[27237]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-20T08:42:19.639 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:42:19 smithi194 ceph-mon[27237]: Deploying daemon osd.0 on smithi183 2024-03-20T08:42:20.553 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:42:20 smithi183 ceph-mon[25285]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-20T08:42:20.553 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:42:20 smithi183 ceph-mon[31860]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-20T08:42:20.638 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:42:20 smithi194 ceph-mon[27237]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-20T08:42:22.550 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:42:22 smithi183 ceph-mon[25285]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-20T08:42:22.550 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:42:22 smithi183 ceph-mon[31860]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-20T08:42:22.638 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:42:22 smithi194 ceph-mon[27237]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-20T08:42:24.454 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:42:24 smithi183 ceph-mon[25285]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-20T08:42:24.454 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:42:24 smithi183 ceph-mon[31860]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-20T08:42:24.638 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:42:24 smithi194 ceph-mon[27237]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-20T08:42:25.704 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:42:25 smithi183 ceph-mon[31860]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:42:25.704 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:42:25 smithi183 ceph-mon[31860]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:42:25.704 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:42:25 smithi183 ceph-mon[31860]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-20T08:42:25.704 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:42:25 smithi183 ceph-mon[31860]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-20T08:42:25.704 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:42:25 smithi183 ceph-mon[31860]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-20T08:42:25.705 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:42:25 smithi183 ceph-mon[31860]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:42:25.705 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:42:25 smithi183 ceph-mon[25285]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:42:25.705 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:42:25 smithi183 ceph-mon[25285]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:42:25.705 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:42:25 smithi183 ceph-mon[25285]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-20T08:42:25.705 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:42:25 smithi183 ceph-mon[25285]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-20T08:42:25.705 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:42:25 smithi183 ceph-mon[25285]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-20T08:42:25.705 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:42:25 smithi183 ceph-mon[25285]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:42:25.784 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:42:25 smithi194 ceph-mon[27237]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:42:25.784 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:42:25 smithi194 ceph-mon[27237]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:42:25.784 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:42:25 smithi194 ceph-mon[27237]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-20T08:42:25.784 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:42:25 smithi194 ceph-mon[27237]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-20T08:42:25.785 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:42:25 smithi194 ceph-mon[27237]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-20T08:42:25.785 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:42:25 smithi194 ceph-mon[27237]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:42:26.685 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:42:26 smithi183 ceph-mon[31860]: pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-20T08:42:26.685 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:42:26 smithi183 ceph-mon[31860]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:42:26.685 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:42:26 smithi183 ceph-mon[31860]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:42:26.685 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:42:26 smithi183 ceph-mon[31860]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:42:26.685 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:42:26 smithi183 ceph-mon[25285]: pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-20T08:42:26.686 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:42:26 smithi183 ceph-mon[25285]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:42:26.686 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:42:26 smithi183 ceph-mon[25285]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:42:26.686 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:42:26 smithi183 ceph-mon[25285]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:42:26.699 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:42:26 smithi194 ceph-mon[27237]: pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-20T08:42:26.699 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:42:26 smithi194 ceph-mon[27237]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:42:26.699 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:42:26 smithi194 ceph-mon[27237]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:42:26.699 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:42:26 smithi194 ceph-mon[27237]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:42:27.362 INFO:teuthology.orchestra.run.smithi183.stdout:Created osd(s) 0 on host 'smithi183' 2024-03-20T08:42:28.024 DEBUG:teuthology.orchestra.run.smithi183:osd.0> sudo journalctl -f -n 0 -u ceph-4c3f3eba-e695-11ee-95c9-87774f69a715@osd.0.service 2024-03-20T08:42:28.027 INFO:tasks.cephadm:Deploying osd.1 on smithi183 with /dev/nvme3n1... 2024-03-20T08:42:28.027 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4c3f3eba-e695-11ee-95c9-87774f69a715 -- ceph-volume lvm zap /dev/nvme3n1 2024-03-20T08:42:28.301 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/4c3f3eba-e695-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T08:42:28.632 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:42:28 smithi183 ceph-mon[31860]: pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-20T08:42:28.633 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:42:28 smithi183 ceph-mon[31860]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:42:28.633 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:42:28 smithi183 ceph-mon[31860]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:42:28.633 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:42:28 smithi183 ceph-mon[31860]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-20T08:42:28.633 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:42:28 smithi183 ceph-mon[31860]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-20T08:42:28.633 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:42:28 smithi183 ceph-mon[31860]: Adjusting osd_memory_target on smithi183 to 15040M 2024-03-20T08:42:28.633 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:42:28 smithi183 ceph-mon[31860]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:42:28.633 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:42:28 smithi183 ceph-mon[31860]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-20T08:42:28.633 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:42:28 smithi183 ceph-mon[31860]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-20T08:42:28.633 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:42:28 smithi183 ceph-mon[31860]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:42:28.633 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:42:28 smithi183 ceph-mon[31860]: from='osd.0 [v2:172.21.15.183:6802/1252352917,v1:172.21.15.183:6803/1252352917]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-03-20T08:42:28.633 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:42:28 smithi183 ceph-mon[31860]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-03-20T08:42:28.634 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:42:28 smithi183 ceph-mon[25285]: pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-20T08:42:28.634 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:42:28 smithi183 ceph-mon[25285]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:42:28.634 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:42:28 smithi183 ceph-mon[25285]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:42:28.634 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:42:28 smithi183 ceph-mon[25285]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-20T08:42:28.634 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:42:28 smithi183 ceph-mon[25285]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-20T08:42:28.634 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:42:28 smithi183 ceph-mon[25285]: Adjusting osd_memory_target on smithi183 to 15040M 2024-03-20T08:42:28.634 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:42:28 smithi183 ceph-mon[25285]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:42:28.634 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:42:28 smithi183 ceph-mon[25285]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-20T08:42:28.634 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:42:28 smithi183 ceph-mon[25285]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-20T08:42:28.634 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:42:28 smithi183 ceph-mon[25285]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:42:28.634 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:42:28 smithi183 ceph-mon[25285]: from='osd.0 [v2:172.21.15.183:6802/1252352917,v1:172.21.15.183:6803/1252352917]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-03-20T08:42:28.634 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:42:28 smithi183 ceph-mon[25285]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-03-20T08:42:28.638 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:42:28 smithi194 ceph-mon[27237]: pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-20T08:42:28.638 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:42:28 smithi194 ceph-mon[27237]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:42:28.638 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:42:28 smithi194 ceph-mon[27237]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:42:28.638 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:42:28 smithi194 ceph-mon[27237]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-20T08:42:28.639 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:42:28 smithi194 ceph-mon[27237]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-20T08:42:28.639 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:42:28 smithi194 ceph-mon[27237]: Adjusting osd_memory_target on smithi183 to 15040M 2024-03-20T08:42:28.639 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:42:28 smithi194 ceph-mon[27237]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:42:28.639 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:42:28 smithi194 ceph-mon[27237]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-20T08:42:28.639 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:42:28 smithi194 ceph-mon[27237]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-20T08:42:28.639 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:42:28 smithi194 ceph-mon[27237]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:42:28.639 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:42:28 smithi194 ceph-mon[27237]: from='osd.0 [v2:172.21.15.183:6802/1252352917,v1:172.21.15.183:6803/1252352917]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-03-20T08:42:28.639 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:42:28 smithi194 ceph-mon[27237]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-03-20T08:42:29.664 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:42:29 smithi183 ceph-mon[25285]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-03-20T08:42:29.664 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:42:29 smithi183 ceph-mon[25285]: osdmap e6: 1 total, 0 up, 1 in 2024-03-20T08:42:29.664 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:42:29 smithi183 ceph-mon[25285]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-20T08:42:29.664 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:42:29 smithi183 ceph-mon[25285]: from='osd.0 [v2:172.21.15.183:6802/1252352917,v1:172.21.15.183:6803/1252352917]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi183", "root=default"]}]: dispatch 2024-03-20T08:42:29.664 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:42:29 smithi183 ceph-mon[25285]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi183", "root=default"]}]: dispatch 2024-03-20T08:42:29.664 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:42:29 smithi183 ceph-mon[31860]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-03-20T08:42:29.664 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:42:29 smithi183 ceph-mon[31860]: osdmap e6: 1 total, 0 up, 1 in 2024-03-20T08:42:29.664 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:42:29 smithi183 ceph-mon[31860]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-20T08:42:29.665 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:42:29 smithi183 ceph-mon[31860]: from='osd.0 [v2:172.21.15.183:6802/1252352917,v1:172.21.15.183:6803/1252352917]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi183", "root=default"]}]: dispatch 2024-03-20T08:42:29.665 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:42:29 smithi183 ceph-mon[31860]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi183", "root=default"]}]: dispatch 2024-03-20T08:42:29.665 INFO:journalctl@ceph.osd.0.smithi183.stdout:Mar 20 08:42:29 smithi183 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-osd-0[36374]: 2024-03-20T08:42:29.428+0000 7f89c4954700 -1 osd.0 0 waiting for initial osdmap 2024-03-20T08:42:29.665 INFO:journalctl@ceph.osd.0.smithi183.stdout:Mar 20 08:42:29 smithi183 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-osd-0[36374]: 2024-03-20T08:42:29.430+0000 7f89c0747700 -1 osd.0 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-20T08:42:29.888 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:42:29 smithi194 ceph-mon[27237]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-03-20T08:42:29.888 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:42:29 smithi194 ceph-mon[27237]: osdmap e6: 1 total, 0 up, 1 in 2024-03-20T08:42:29.889 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:42:29 smithi194 ceph-mon[27237]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-20T08:42:29.889 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:42:29 smithi194 ceph-mon[27237]: from='osd.0 [v2:172.21.15.183:6802/1252352917,v1:172.21.15.183:6803/1252352917]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi183", "root=default"]}]: dispatch 2024-03-20T08:42:29.889 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:42:29 smithi194 ceph-mon[27237]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi183", "root=default"]}]: dispatch 2024-03-20T08:42:30.099 INFO:teuthology.orchestra.run.smithi183.stderr:--> Zapping: /dev/nvme3n1 2024-03-20T08:42:30.099 INFO:teuthology.orchestra.run.smithi183.stderr:--> --destroy was not specified, but zapping a whole device will remove the partition table 2024-03-20T08:42:30.099 INFO:teuthology.orchestra.run.smithi183.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/nvme3n1 bs=1M count=10 conv=fsync 2024-03-20T08:42:30.099 INFO:teuthology.orchestra.run.smithi183.stderr: stderr: 10+0 records in 2024-03-20T08:42:30.099 INFO:teuthology.orchestra.run.smithi183.stderr:10+0 records out 2024-03-20T08:42:30.099 INFO:teuthology.orchestra.run.smithi183.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.013776 s, 761 MB/s 2024-03-20T08:42:30.099 INFO:teuthology.orchestra.run.smithi183.stderr:--> Zapping successful for: 2024-03-20T08:42:30.704 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:42:30 smithi183 ceph-mon[31860]: pgmap v42: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-20T08:42:30.704 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:42:30 smithi183 ceph-mon[31860]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi183", "root=default"]}]': finished 2024-03-20T08:42:30.704 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:42:30 smithi183 ceph-mon[31860]: osdmap e7: 1 total, 0 up, 1 in 2024-03-20T08:42:30.704 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:42:30 smithi183 ceph-mon[31860]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-20T08:42:30.704 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:42:30 smithi183 ceph-mon[31860]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-20T08:42:30.705 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:42:30 smithi183 ceph-mon[31860]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-20T08:42:30.705 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:42:30 smithi183 ceph-mon[25285]: pgmap v42: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-20T08:42:30.705 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:42:30 smithi183 ceph-mon[25285]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi183", "root=default"]}]': finished 2024-03-20T08:42:30.705 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:42:30 smithi183 ceph-mon[25285]: osdmap e7: 1 total, 0 up, 1 in 2024-03-20T08:42:30.705 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:42:30 smithi183 ceph-mon[25285]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-20T08:42:30.705 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:42:30 smithi183 ceph-mon[25285]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-20T08:42:30.705 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:42:30 smithi183 ceph-mon[25285]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-20T08:42:30.779 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4c3f3eba-e695-11ee-95c9-87774f69a715 -- ceph orch daemon add osd smithi183:/dev/nvme3n1 2024-03-20T08:42:30.888 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:42:30 smithi194 ceph-mon[27237]: pgmap v42: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-20T08:42:30.888 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:42:30 smithi194 ceph-mon[27237]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi183", "root=default"]}]': finished 2024-03-20T08:42:30.888 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:42:30 smithi194 ceph-mon[27237]: osdmap e7: 1 total, 0 up, 1 in 2024-03-20T08:42:30.889 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:42:30 smithi194 ceph-mon[27237]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-20T08:42:30.889 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:42:30 smithi194 ceph-mon[27237]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-20T08:42:30.889 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:42:30 smithi194 ceph-mon[27237]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-20T08:42:31.016 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/4c3f3eba-e695-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T08:42:31.704 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:42:31 smithi183 ceph-mon[25285]: purged_snaps scrub starts 2024-03-20T08:42:31.704 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:42:31 smithi183 ceph-mon[25285]: purged_snaps scrub ok 2024-03-20T08:42:31.704 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:42:31 smithi183 ceph-mon[25285]: osd.0 [v2:172.21.15.183:6802/1252352917,v1:172.21.15.183:6803/1252352917] boot 2024-03-20T08:42:31.704 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:42:31 smithi183 ceph-mon[25285]: osdmap e8: 1 total, 1 up, 1 in 2024-03-20T08:42:31.704 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:42:31 smithi183 ceph-mon[25285]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-20T08:42:31.705 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:42:31 smithi183 ceph-mon[31860]: purged_snaps scrub starts 2024-03-20T08:42:31.705 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:42:31 smithi183 ceph-mon[31860]: purged_snaps scrub ok 2024-03-20T08:42:31.705 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:42:31 smithi183 ceph-mon[31860]: osd.0 [v2:172.21.15.183:6802/1252352917,v1:172.21.15.183:6803/1252352917] boot 2024-03-20T08:42:31.705 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:42:31 smithi183 ceph-mon[31860]: osdmap e8: 1 total, 1 up, 1 in 2024-03-20T08:42:31.705 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:42:31 smithi183 ceph-mon[31860]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-20T08:42:31.890 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:42:31 smithi194 ceph-mon[27237]: purged_snaps scrub starts 2024-03-20T08:42:31.891 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:42:31 smithi194 ceph-mon[27237]: purged_snaps scrub ok 2024-03-20T08:42:31.891 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:42:31 smithi194 ceph-mon[27237]: osd.0 [v2:172.21.15.183:6802/1252352917,v1:172.21.15.183:6803/1252352917] boot 2024-03-20T08:42:31.891 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:42:31 smithi194 ceph-mon[27237]: osdmap e8: 1 total, 1 up, 1 in 2024-03-20T08:42:31.891 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:42:31 smithi194 ceph-mon[27237]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-20T08:42:32.706 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:42:32 smithi183 ceph-mon[25285]: pgmap v45: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-20T08:42:32.706 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:42:32 smithi183 ceph-mon[25285]: osdmap e9: 1 total, 1 up, 1 in 2024-03-20T08:42:32.707 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:42:32 smithi183 ceph-mon[31860]: pgmap v45: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-20T08:42:32.707 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:42:32 smithi183 ceph-mon[31860]: osdmap e9: 1 total, 1 up, 1 in 2024-03-20T08:42:32.888 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:42:32 smithi194 ceph-mon[27237]: pgmap v45: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-20T08:42:32.888 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:42:32 smithi194 ceph-mon[27237]: osdmap e9: 1 total, 1 up, 1 in 2024-03-20T08:42:33.695 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:42:33 smithi183 ceph-mon[25285]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-20T08:42:33.695 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:42:33 smithi183 ceph-mon[25285]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-20T08:42:33.696 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:42:33 smithi183 ceph-mon[25285]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-20T08:42:33.696 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:42:33 smithi183 ceph-mon[31860]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-20T08:42:33.696 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:42:33 smithi183 ceph-mon[31860]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-20T08:42:33.696 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:42:33 smithi183 ceph-mon[31860]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-20T08:42:33.888 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:42:33 smithi194 ceph-mon[27237]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-20T08:42:33.888 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:42:33 smithi194 ceph-mon[27237]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-20T08:42:33.888 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:42:33 smithi194 ceph-mon[27237]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-20T08:42:34.704 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:42:34 smithi183 ceph-mon[31860]: pgmap v47: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-20T08:42:34.704 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:42:34 smithi183 ceph-mon[31860]: from='client.14244 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi183:/dev/nvme3n1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-20T08:42:34.705 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:42:34 smithi183 ceph-mon[25285]: pgmap v47: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-20T08:42:34.705 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:42:34 smithi183 ceph-mon[25285]: from='client.14244 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi183:/dev/nvme3n1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-20T08:42:34.888 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:42:34 smithi194 ceph-mon[27237]: pgmap v47: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-20T08:42:34.888 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:42:34 smithi194 ceph-mon[27237]: from='client.14244 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi183:/dev/nvme3n1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-20T08:42:36.954 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:42:36 smithi183 ceph-mon[25285]: pgmap v48: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-20T08:42:36.955 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:42:36 smithi183 ceph-mon[25285]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:42:36.955 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:42:36 smithi183 ceph-mon[25285]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:42:36.955 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:42:36 smithi183 ceph-mon[25285]: Detected new or changed devices on smithi183 2024-03-20T08:42:36.955 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:42:36 smithi183 ceph-mon[25285]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:42:36.955 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:42:36 smithi183 ceph-mon[25285]: from='client.? 172.21.15.183:0/2073851110' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c881374c-df10-42e3-8a63-3f06638b1d44"}]: dispatch 2024-03-20T08:42:36.955 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:42:36 smithi183 ceph-mon[25285]: from='client.? 172.21.15.183:0/2073851110' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "c881374c-df10-42e3-8a63-3f06638b1d44"}]': finished 2024-03-20T08:42:36.955 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:42:36 smithi183 ceph-mon[25285]: osdmap e10: 2 total, 1 up, 2 in 2024-03-20T08:42:36.955 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:42:36 smithi183 ceph-mon[25285]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-20T08:42:36.955 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:42:36 smithi183 ceph-mon[31860]: pgmap v48: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-20T08:42:36.955 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:42:36 smithi183 ceph-mon[31860]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:42:36.955 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:42:36 smithi183 ceph-mon[31860]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:42:36.955 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:42:36 smithi183 ceph-mon[31860]: Detected new or changed devices on smithi183 2024-03-20T08:42:36.956 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:42:36 smithi183 ceph-mon[31860]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:42:36.956 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:42:36 smithi183 ceph-mon[31860]: from='client.? 172.21.15.183:0/2073851110' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c881374c-df10-42e3-8a63-3f06638b1d44"}]: dispatch 2024-03-20T08:42:36.956 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:42:36 smithi183 ceph-mon[31860]: from='client.? 172.21.15.183:0/2073851110' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "c881374c-df10-42e3-8a63-3f06638b1d44"}]': finished 2024-03-20T08:42:36.956 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:42:36 smithi183 ceph-mon[31860]: osdmap e10: 2 total, 1 up, 2 in 2024-03-20T08:42:36.956 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:42:36 smithi183 ceph-mon[31860]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-20T08:42:37.138 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:42:36 smithi194 ceph-mon[27237]: pgmap v48: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-20T08:42:37.139 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:42:36 smithi194 ceph-mon[27237]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:42:37.139 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:42:36 smithi194 ceph-mon[27237]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:42:37.139 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:42:36 smithi194 ceph-mon[27237]: Detected new or changed devices on smithi183 2024-03-20T08:42:37.139 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:42:36 smithi194 ceph-mon[27237]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:42:37.139 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:42:36 smithi194 ceph-mon[27237]: from='client.? 172.21.15.183:0/2073851110' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c881374c-df10-42e3-8a63-3f06638b1d44"}]: dispatch 2024-03-20T08:42:37.139 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:42:36 smithi194 ceph-mon[27237]: from='client.? 172.21.15.183:0/2073851110' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "c881374c-df10-42e3-8a63-3f06638b1d44"}]': finished 2024-03-20T08:42:37.139 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:42:36 smithi194 ceph-mon[27237]: osdmap e10: 2 total, 1 up, 2 in 2024-03-20T08:42:37.139 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:42:36 smithi194 ceph-mon[27237]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-20T08:42:37.954 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:42:37 smithi183 ceph-mon[25285]: from='client.? 172.21.15.183:0/3192917594' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-20T08:42:37.955 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:42:37 smithi183 ceph-mon[31860]: from='client.? 172.21.15.183:0/3192917594' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-20T08:42:38.138 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:42:37 smithi194 ceph-mon[27237]: from='client.? 172.21.15.183:0/3192917594' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-20T08:42:38.954 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:42:38 smithi183 ceph-mon[25285]: pgmap v50: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-20T08:42:38.954 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:42:38 smithi183 ceph-mon[31860]: pgmap v50: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-20T08:42:39.138 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:42:38 smithi194 ceph-mon[27237]: pgmap v50: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-20T08:42:40.834 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:42:40 smithi183 ceph-mon[25285]: pgmap v51: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-20T08:42:40.834 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:42:40 smithi183 ceph-mon[31860]: pgmap v51: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-20T08:42:40.940 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:42:40 smithi194 ceph-mon[27237]: pgmap v51: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-20T08:42:42.861 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:42:42 smithi183 ceph-mon[25285]: pgmap v52: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-20T08:42:42.862 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:42:42 smithi183 ceph-mon[31860]: pgmap v52: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-20T08:42:43.138 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:42:42 smithi194 ceph-mon[27237]: pgmap v52: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-20T08:42:44.954 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:42:44 smithi183 ceph-mon[25285]: pgmap v53: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-20T08:42:44.954 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:42:44 smithi183 ceph-mon[25285]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-03-20T08:42:44.954 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:42:44 smithi183 ceph-mon[25285]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-20T08:42:44.955 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:42:44 smithi183 ceph-mon[31860]: pgmap v53: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-20T08:42:44.955 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:42:44 smithi183 ceph-mon[31860]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-03-20T08:42:44.955 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:42:44 smithi183 ceph-mon[31860]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-20T08:42:44.979 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:42:44 smithi194 ceph-mon[27237]: pgmap v53: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-20T08:42:44.979 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:42:44 smithi194 ceph-mon[27237]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-03-20T08:42:44.979 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:42:44 smithi194 ceph-mon[27237]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-20T08:42:45.954 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:42:45 smithi183 ceph-mon[25285]: Deploying daemon osd.1 on smithi183 2024-03-20T08:42:45.955 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:42:45 smithi183 ceph-mon[31860]: Deploying daemon osd.1 on smithi183 2024-03-20T08:42:46.138 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:42:45 smithi194 ceph-mon[27237]: Deploying daemon osd.1 on smithi183 2024-03-20T08:42:46.686 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:42:46 smithi194 ceph-mon[27237]: pgmap v54: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-20T08:42:46.686 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:42:46 smithi194 ceph-mon[27237]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:42:46.942 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:42:46 smithi183 ceph-mon[31860]: pgmap v54: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-20T08:42:46.943 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:42:46 smithi183 ceph-mon[31860]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:42:46.943 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:42:46 smithi183 ceph-mon[31860]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:42:46.943 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:42:46 smithi183 ceph-mon[31860]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:42:46.943 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:42:46 smithi183 ceph-mon[25285]: pgmap v54: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-20T08:42:46.943 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:42:46 smithi183 ceph-mon[25285]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:42:46.943 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:42:46 smithi183 ceph-mon[25285]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:42:46.943 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:42:46 smithi183 ceph-mon[25285]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:42:47.138 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:42:46 smithi194 ceph-mon[27237]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:42:47.138 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:42:46 smithi194 ceph-mon[27237]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:42:48.945 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:42:48 smithi183 ceph-mon[25285]: pgmap v55: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-20T08:42:48.945 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:42:48 smithi183 ceph-mon[31860]: pgmap v55: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-20T08:42:49.138 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:42:48 smithi194 ceph-mon[27237]: pgmap v55: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-20T08:42:50.638 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:42:50 smithi194 ceph-mon[27237]: pgmap v56: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-20T08:42:50.639 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:42:50 smithi194 ceph-mon[27237]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:42:50.639 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:42:50 smithi194 ceph-mon[27237]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:42:50.639 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:42:50 smithi194 ceph-mon[27237]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-20T08:42:50.639 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:42:50 smithi194 ceph-mon[27237]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-20T08:42:50.639 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:42:50 smithi194 ceph-mon[27237]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-20T08:42:50.639 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:42:50 smithi194 ceph-mon[27237]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:42:50.660 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:42:50 smithi183 ceph-mon[25285]: pgmap v56: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-20T08:42:50.660 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:42:50 smithi183 ceph-mon[25285]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:42:50.660 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:42:50 smithi183 ceph-mon[25285]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:42:50.660 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:42:50 smithi183 ceph-mon[25285]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-20T08:42:50.660 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:42:50 smithi183 ceph-mon[25285]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-20T08:42:50.660 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:42:50 smithi183 ceph-mon[25285]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-20T08:42:50.660 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:42:50 smithi183 ceph-mon[25285]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:42:50.660 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:42:50 smithi183 ceph-mon[31860]: pgmap v56: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-20T08:42:50.661 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:42:50 smithi183 ceph-mon[31860]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:42:50.661 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:42:50 smithi183 ceph-mon[31860]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:42:50.661 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:42:50 smithi183 ceph-mon[31860]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-20T08:42:50.661 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:42:50 smithi183 ceph-mon[31860]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-20T08:42:50.661 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:42:50 smithi183 ceph-mon[31860]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-20T08:42:50.661 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:42:50 smithi183 ceph-mon[31860]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:42:52.415 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:42:52 smithi183 ceph-mon[25285]: pgmap v57: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-20T08:42:52.415 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:42:52 smithi183 ceph-mon[31860]: pgmap v57: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-20T08:42:52.596 INFO:teuthology.orchestra.run.smithi183.stdout:Created osd(s) 1 on host 'smithi183' 2024-03-20T08:42:52.638 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:42:52 smithi194 ceph-mon[27237]: pgmap v57: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-20T08:42:53.191 DEBUG:teuthology.orchestra.run.smithi183:osd.1> sudo journalctl -f -n 0 -u ceph-4c3f3eba-e695-11ee-95c9-87774f69a715@osd.1.service 2024-03-20T08:42:53.193 INFO:tasks.cephadm:Deploying osd.2 on smithi183 with /dev/nvme2n1... 2024-03-20T08:42:53.194 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4c3f3eba-e695-11ee-95c9-87774f69a715 -- ceph-volume lvm zap /dev/nvme2n1 2024-03-20T08:42:53.469 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/4c3f3eba-e695-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T08:42:53.590 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:42:53 smithi183 ceph-mon[25285]: from='osd.1 [v2:172.21.15.183:6810/3442013001,v1:172.21.15.183:6811/3442013001]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-20T08:42:53.590 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:42:53 smithi183 ceph-mon[25285]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:42:53.590 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:42:53 smithi183 ceph-mon[25285]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:42:53.590 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:42:53 smithi183 ceph-mon[25285]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-20T08:42:53.590 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:42:53 smithi183 ceph-mon[25285]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-20T08:42:53.590 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:42:53 smithi183 ceph-mon[25285]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-20T08:42:53.590 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:42:53 smithi183 ceph-mon[25285]: Adjusting osd_memory_target on smithi183 to 7520M 2024-03-20T08:42:53.590 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:42:53 smithi183 ceph-mon[25285]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:42:53.590 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:42:53 smithi183 ceph-mon[25285]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-20T08:42:53.591 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:42:53 smithi183 ceph-mon[25285]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-20T08:42:53.591 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:42:53 smithi183 ceph-mon[25285]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:42:53.591 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:42:53 smithi183 ceph-mon[31860]: from='osd.1 [v2:172.21.15.183:6810/3442013001,v1:172.21.15.183:6811/3442013001]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-20T08:42:53.591 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:42:53 smithi183 ceph-mon[31860]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:42:53.591 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:42:53 smithi183 ceph-mon[31860]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:42:53.591 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:42:53 smithi183 ceph-mon[31860]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-20T08:42:53.591 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:42:53 smithi183 ceph-mon[31860]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-20T08:42:53.591 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:42:53 smithi183 ceph-mon[31860]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-20T08:42:53.591 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:42:53 smithi183 ceph-mon[31860]: Adjusting osd_memory_target on smithi183 to 7520M 2024-03-20T08:42:53.591 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:42:53 smithi183 ceph-mon[31860]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:42:53.591 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:42:53 smithi183 ceph-mon[31860]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-20T08:42:53.591 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:42:53 smithi183 ceph-mon[31860]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-20T08:42:53.591 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:42:53 smithi183 ceph-mon[31860]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:42:53.638 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:42:53 smithi194 ceph-mon[27237]: from='osd.1 [v2:172.21.15.183:6810/3442013001,v1:172.21.15.183:6811/3442013001]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-20T08:42:53.639 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:42:53 smithi194 ceph-mon[27237]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:42:53.639 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:42:53 smithi194 ceph-mon[27237]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:42:53.639 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:42:53 smithi194 ceph-mon[27237]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-20T08:42:53.639 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:42:53 smithi194 ceph-mon[27237]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-20T08:42:53.639 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:42:53 smithi194 ceph-mon[27237]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-20T08:42:53.639 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:42:53 smithi194 ceph-mon[27237]: Adjusting osd_memory_target on smithi183 to 7520M 2024-03-20T08:42:53.639 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:42:53 smithi194 ceph-mon[27237]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:42:53.639 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:42:53 smithi194 ceph-mon[27237]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-20T08:42:53.639 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:42:53 smithi194 ceph-mon[27237]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-20T08:42:53.639 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:42:53 smithi194 ceph-mon[27237]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:42:54.519 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:42:54 smithi183 ceph-mon[25285]: pgmap v58: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-20T08:42:54.519 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:42:54 smithi183 ceph-mon[25285]: from='osd.1 [v2:172.21.15.183:6810/3442013001,v1:172.21.15.183:6811/3442013001]' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-03-20T08:42:54.519 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:42:54 smithi183 ceph-mon[25285]: osdmap e11: 2 total, 1 up, 2 in 2024-03-20T08:42:54.520 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:42:54 smithi183 ceph-mon[25285]: from='osd.1 [v2:172.21.15.183:6810/3442013001,v1:172.21.15.183:6811/3442013001]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi183", "root=default"]}]: dispatch 2024-03-20T08:42:54.520 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:42:54 smithi183 ceph-mon[25285]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-20T08:42:54.520 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:42:54 smithi183 ceph-mon[31860]: pgmap v58: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-20T08:42:54.520 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:42:54 smithi183 ceph-mon[31860]: from='osd.1 [v2:172.21.15.183:6810/3442013001,v1:172.21.15.183:6811/3442013001]' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-03-20T08:42:54.520 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:42:54 smithi183 ceph-mon[31860]: osdmap e11: 2 total, 1 up, 2 in 2024-03-20T08:42:54.521 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:42:54 smithi183 ceph-mon[31860]: from='osd.1 [v2:172.21.15.183:6810/3442013001,v1:172.21.15.183:6811/3442013001]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi183", "root=default"]}]: dispatch 2024-03-20T08:42:54.521 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:42:54 smithi183 ceph-mon[31860]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-20T08:42:54.638 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:42:54 smithi194 ceph-mon[27237]: pgmap v58: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-20T08:42:54.639 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:42:54 smithi194 ceph-mon[27237]: from='osd.1 [v2:172.21.15.183:6810/3442013001,v1:172.21.15.183:6811/3442013001]' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-03-20T08:42:54.639 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:42:54 smithi194 ceph-mon[27237]: osdmap e11: 2 total, 1 up, 2 in 2024-03-20T08:42:54.639 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:42:54 smithi194 ceph-mon[27237]: from='osd.1 [v2:172.21.15.183:6810/3442013001,v1:172.21.15.183:6811/3442013001]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi183", "root=default"]}]: dispatch 2024-03-20T08:42:54.639 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:42:54 smithi194 ceph-mon[27237]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-20T08:42:54.788 INFO:journalctl@ceph.osd.1.smithi183.stdout:Mar 20 08:42:54 smithi183 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-osd-1[39792]: 2024-03-20T08:42:54.745+0000 7f8e96dc1700 -1 osd.1 0 waiting for initial osdmap 2024-03-20T08:42:54.788 INFO:journalctl@ceph.osd.1.smithi183.stdout:Mar 20 08:42:54 smithi183 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-osd-1[39792]: 2024-03-20T08:42:54.749+0000 7f8e8e3ab700 -1 osd.1 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-20T08:42:55.346 INFO:teuthology.orchestra.run.smithi183.stderr:--> Zapping: /dev/nvme2n1 2024-03-20T08:42:55.346 INFO:teuthology.orchestra.run.smithi183.stderr:--> --destroy was not specified, but zapping a whole device will remove the partition table 2024-03-20T08:42:55.346 INFO:teuthology.orchestra.run.smithi183.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/nvme2n1 bs=1M count=10 conv=fsync 2024-03-20T08:42:55.346 INFO:teuthology.orchestra.run.smithi183.stderr: stderr: 10+0 records in 2024-03-20T08:42:55.346 INFO:teuthology.orchestra.run.smithi183.stderr:10+0 records out 2024-03-20T08:42:55.346 INFO:teuthology.orchestra.run.smithi183.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0191369 s, 548 MB/s 2024-03-20T08:42:55.346 INFO:teuthology.orchestra.run.smithi183.stderr:--> Zapping successful for: 2024-03-20T08:42:55.638 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:42:55 smithi194 ceph-mon[27237]: from='osd.1 [v2:172.21.15.183:6810/3442013001,v1:172.21.15.183:6811/3442013001]' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi183", "root=default"]}]': finished 2024-03-20T08:42:55.639 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:42:55 smithi194 ceph-mon[27237]: osdmap e12: 2 total, 1 up, 2 in 2024-03-20T08:42:55.639 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:42:55 smithi194 ceph-mon[27237]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-20T08:42:55.639 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:42:55 smithi194 ceph-mon[27237]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-20T08:42:55.639 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:42:55 smithi194 ceph-mon[27237]: from='osd.1 [v2:172.21.15.183:6810/3442013001,v1:172.21.15.183:6811/3442013001]' entity='osd.1' 2024-03-20T08:42:55.644 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:42:55 smithi183 ceph-mon[31860]: from='osd.1 [v2:172.21.15.183:6810/3442013001,v1:172.21.15.183:6811/3442013001]' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi183", "root=default"]}]': finished 2024-03-20T08:42:55.644 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:42:55 smithi183 ceph-mon[31860]: osdmap e12: 2 total, 1 up, 2 in 2024-03-20T08:42:55.644 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:42:55 smithi183 ceph-mon[31860]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-20T08:42:55.645 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:42:55 smithi183 ceph-mon[31860]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-20T08:42:55.645 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:42:55 smithi183 ceph-mon[31860]: from='osd.1 [v2:172.21.15.183:6810/3442013001,v1:172.21.15.183:6811/3442013001]' entity='osd.1' 2024-03-20T08:42:55.645 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:42:55 smithi183 ceph-mon[25285]: from='osd.1 [v2:172.21.15.183:6810/3442013001,v1:172.21.15.183:6811/3442013001]' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi183", "root=default"]}]': finished 2024-03-20T08:42:55.645 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:42:55 smithi183 ceph-mon[25285]: osdmap e12: 2 total, 1 up, 2 in 2024-03-20T08:42:55.645 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:42:55 smithi183 ceph-mon[25285]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-20T08:42:55.645 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:42:55 smithi183 ceph-mon[25285]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-20T08:42:55.645 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:42:55 smithi183 ceph-mon[25285]: from='osd.1 [v2:172.21.15.183:6810/3442013001,v1:172.21.15.183:6811/3442013001]' entity='osd.1' 2024-03-20T08:42:56.364 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4c3f3eba-e695-11ee-95c9-87774f69a715 -- ceph orch daemon add osd smithi183:/dev/nvme2n1 2024-03-20T08:42:56.628 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/4c3f3eba-e695-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T08:42:56.638 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:42:56 smithi194 ceph-mon[27237]: purged_snaps scrub starts 2024-03-20T08:42:56.639 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:42:56 smithi194 ceph-mon[27237]: purged_snaps scrub ok 2024-03-20T08:42:56.639 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:42:56 smithi194 ceph-mon[27237]: pgmap v61: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-20T08:42:56.639 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:42:56 smithi194 ceph-mon[27237]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-20T08:42:56.639 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:42:56 smithi194 ceph-mon[27237]: osd.1 [v2:172.21.15.183:6810/3442013001,v1:172.21.15.183:6811/3442013001] boot 2024-03-20T08:42:56.639 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:42:56 smithi194 ceph-mon[27237]: osdmap e13: 2 total, 2 up, 2 in 2024-03-20T08:42:56.639 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:42:56 smithi194 ceph-mon[27237]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-20T08:42:56.639 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:42:56 smithi194 ceph-mon[27237]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:42:56.639 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:42:56 smithi194 ceph-mon[27237]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:42:56.639 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:42:56 smithi194 ceph-mon[27237]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:42:56.639 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:42:56 smithi194 ceph-mon[27237]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-20T08:42:56.640 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:42:56 smithi194 ceph-mon[27237]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-20T08:42:56.640 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:42:56 smithi194 ceph-mon[27237]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-20T08:42:56.640 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:42:56 smithi194 ceph-mon[27237]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:42:56.704 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:42:56 smithi183 ceph-mon[25285]: purged_snaps scrub starts 2024-03-20T08:42:56.705 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:42:56 smithi183 ceph-mon[25285]: purged_snaps scrub ok 2024-03-20T08:42:56.705 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:42:56 smithi183 ceph-mon[25285]: pgmap v61: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-20T08:42:56.705 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:42:56 smithi183 ceph-mon[25285]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-20T08:42:56.705 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:42:56 smithi183 ceph-mon[25285]: osd.1 [v2:172.21.15.183:6810/3442013001,v1:172.21.15.183:6811/3442013001] boot 2024-03-20T08:42:56.705 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:42:56 smithi183 ceph-mon[25285]: osdmap e13: 2 total, 2 up, 2 in 2024-03-20T08:42:56.705 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:42:56 smithi183 ceph-mon[25285]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-20T08:42:56.705 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:42:56 smithi183 ceph-mon[25285]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:42:56.705 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:42:56 smithi183 ceph-mon[25285]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:42:56.705 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:42:56 smithi183 ceph-mon[25285]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:42:56.705 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:42:56 smithi183 ceph-mon[25285]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-20T08:42:56.705 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:42:56 smithi183 ceph-mon[25285]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-20T08:42:56.705 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:42:56 smithi183 ceph-mon[25285]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-20T08:42:56.706 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:42:56 smithi183 ceph-mon[25285]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:42:56.706 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:42:56 smithi183 ceph-mon[31860]: purged_snaps scrub starts 2024-03-20T08:42:56.706 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:42:56 smithi183 ceph-mon[31860]: purged_snaps scrub ok 2024-03-20T08:42:56.706 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:42:56 smithi183 ceph-mon[31860]: pgmap v61: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-20T08:42:56.706 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:42:56 smithi183 ceph-mon[31860]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-20T08:42:56.706 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:42:56 smithi183 ceph-mon[31860]: osd.1 [v2:172.21.15.183:6810/3442013001,v1:172.21.15.183:6811/3442013001] boot 2024-03-20T08:42:56.706 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:42:56 smithi183 ceph-mon[31860]: osdmap e13: 2 total, 2 up, 2 in 2024-03-20T08:42:56.706 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:42:56 smithi183 ceph-mon[31860]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-20T08:42:56.706 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:42:56 smithi183 ceph-mon[31860]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:42:56.706 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:42:56 smithi183 ceph-mon[31860]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:42:56.706 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:42:56 smithi183 ceph-mon[31860]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:42:56.707 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:42:56 smithi183 ceph-mon[31860]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-20T08:42:56.707 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:42:56 smithi183 ceph-mon[31860]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-20T08:42:56.707 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:42:56 smithi183 ceph-mon[31860]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-20T08:42:56.707 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:42:56 smithi183 ceph-mon[31860]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:42:57.426 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:42:57 smithi183 ceph-mon[25285]: Detected new or changed devices on smithi183 2024-03-20T08:42:57.426 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:42:57 smithi183 ceph-mon[25285]: pgmap v63: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-20T08:42:57.426 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:42:57 smithi183 ceph-mon[25285]: Health check failed: 1 stray daemon(s) not managed by cephadm (CEPHADM_STRAY_DAEMON) 2024-03-20T08:42:57.427 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:42:57 smithi183 ceph-mon[31860]: Detected new or changed devices on smithi183 2024-03-20T08:42:57.427 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:42:57 smithi183 ceph-mon[31860]: pgmap v63: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-20T08:42:57.427 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:42:57 smithi183 ceph-mon[31860]: Health check failed: 1 stray daemon(s) not managed by cephadm (CEPHADM_STRAY_DAEMON) 2024-03-20T08:42:57.638 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:42:57 smithi194 ceph-mon[27237]: Detected new or changed devices on smithi183 2024-03-20T08:42:57.638 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:42:57 smithi194 ceph-mon[27237]: pgmap v63: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-20T08:42:57.639 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:42:57 smithi194 ceph-mon[27237]: Health check failed: 1 stray daemon(s) not managed by cephadm (CEPHADM_STRAY_DAEMON) 2024-03-20T08:42:58.574 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:42:58 smithi183 ceph-mon[25285]: osdmap e14: 2 total, 2 up, 2 in 2024-03-20T08:42:58.574 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:42:58 smithi183 ceph-mon[31860]: osdmap e14: 2 total, 2 up, 2 in 2024-03-20T08:42:58.638 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:42:58 smithi194 ceph-mon[27237]: osdmap e14: 2 total, 2 up, 2 in 2024-03-20T08:42:59.455 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:42:59 smithi183 ceph-mon[25285]: pgmap v65: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-20T08:42:59.455 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:42:59 smithi183 ceph-mon[25285]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:42:59.455 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:42:59 smithi183 ceph-mon[25285]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:42:59.455 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:42:59 smithi183 ceph-mon[25285]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-20T08:42:59.455 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:42:59 smithi183 ceph-mon[25285]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-20T08:42:59.455 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:42:59 smithi183 ceph-mon[25285]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:42:59.455 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:42:59 smithi183 ceph-mon[25285]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-20T08:42:59.455 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:42:59 smithi183 ceph-mon[25285]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:42:59.455 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:42:59 smithi183 ceph-mon[31860]: pgmap v65: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-20T08:42:59.456 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:42:59 smithi183 ceph-mon[31860]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:42:59.456 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:42:59 smithi183 ceph-mon[31860]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:42:59.456 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:42:59 smithi183 ceph-mon[31860]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-20T08:42:59.456 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:42:59 smithi183 ceph-mon[31860]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-20T08:42:59.456 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:42:59 smithi183 ceph-mon[31860]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:42:59.456 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:42:59 smithi183 ceph-mon[31860]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-20T08:42:59.456 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:42:59 smithi183 ceph-mon[31860]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:42:59.638 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:42:59 smithi194 ceph-mon[27237]: pgmap v65: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-20T08:42:59.639 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:42:59 smithi194 ceph-mon[27237]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:42:59.639 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:42:59 smithi194 ceph-mon[27237]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:42:59.639 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:42:59 smithi194 ceph-mon[27237]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-20T08:42:59.639 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:42:59 smithi194 ceph-mon[27237]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-20T08:42:59.639 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:42:59 smithi194 ceph-mon[27237]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:42:59.639 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:42:59 smithi194 ceph-mon[27237]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-20T08:42:59.639 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:42:59 smithi194 ceph-mon[27237]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:43:00.592 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:43:00 smithi183 ceph-mon[25285]: pgmap v66: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-20T08:43:00.592 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:43:00 smithi183 ceph-mon[25285]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-20T08:43:00.592 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:43:00 smithi183 ceph-mon[25285]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-20T08:43:00.592 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:43:00 smithi183 ceph-mon[25285]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-20T08:43:00.592 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:43:00 smithi183 ceph-mon[25285]: Health check cleared: CEPHADM_STRAY_DAEMON (was: 1 stray daemon(s) not managed by cephadm) 2024-03-20T08:43:00.592 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:43:00 smithi183 ceph-mon[25285]: Cluster is now healthy 2024-03-20T08:43:00.593 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:43:00 smithi183 ceph-mon[31860]: pgmap v66: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-20T08:43:00.593 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:43:00 smithi183 ceph-mon[31860]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-20T08:43:00.593 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:43:00 smithi183 ceph-mon[31860]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-20T08:43:00.593 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:43:00 smithi183 ceph-mon[31860]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-20T08:43:00.593 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:43:00 smithi183 ceph-mon[31860]: Health check cleared: CEPHADM_STRAY_DAEMON (was: 1 stray daemon(s) not managed by cephadm) 2024-03-20T08:43:00.593 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:43:00 smithi183 ceph-mon[31860]: Cluster is now healthy 2024-03-20T08:43:00.638 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:43:00 smithi194 ceph-mon[27237]: pgmap v66: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-20T08:43:00.638 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:43:00 smithi194 ceph-mon[27237]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-20T08:43:00.639 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:43:00 smithi194 ceph-mon[27237]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-20T08:43:00.639 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:43:00 smithi194 ceph-mon[27237]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-20T08:43:00.639 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:43:00 smithi194 ceph-mon[27237]: Health check cleared: CEPHADM_STRAY_DAEMON (was: 1 stray daemon(s) not managed by cephadm) 2024-03-20T08:43:00.639 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:43:00 smithi194 ceph-mon[27237]: Cluster is now healthy 2024-03-20T08:43:01.624 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:43:01 smithi183 ceph-mon[25285]: from='client.14271 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi183:/dev/nvme2n1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-20T08:43:01.624 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:43:01 smithi183 ceph-mon[31860]: from='client.14271 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi183:/dev/nvme2n1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-20T08:43:01.638 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:43:01 smithi194 ceph-mon[27237]: from='client.14271 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi183:/dev/nvme2n1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-20T08:43:02.638 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:43:02 smithi194 ceph-mon[27237]: pgmap v67: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-20T08:43:02.704 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:43:02 smithi183 ceph-mon[25285]: pgmap v67: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-20T08:43:02.705 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:43:02 smithi183 ceph-mon[31860]: pgmap v67: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-20T08:43:03.639 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:43:03 smithi194 ceph-mon[27237]: from='client.? 172.21.15.183:0/2159720631' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c1040fce-5e05-4b4f-85b9-da261546233e"}]: dispatch 2024-03-20T08:43:03.639 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:43:03 smithi194 ceph-mon[27237]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c1040fce-5e05-4b4f-85b9-da261546233e"}]: dispatch 2024-03-20T08:43:03.639 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:43:03 smithi194 ceph-mon[27237]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "c1040fce-5e05-4b4f-85b9-da261546233e"}]': finished 2024-03-20T08:43:03.639 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:43:03 smithi194 ceph-mon[27237]: osdmap e15: 3 total, 2 up, 3 in 2024-03-20T08:43:03.639 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:43:03 smithi194 ceph-mon[27237]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-20T08:43:03.704 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:43:03 smithi183 ceph-mon[25285]: from='client.? 172.21.15.183:0/2159720631' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c1040fce-5e05-4b4f-85b9-da261546233e"}]: dispatch 2024-03-20T08:43:03.704 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:43:03 smithi183 ceph-mon[25285]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c1040fce-5e05-4b4f-85b9-da261546233e"}]: dispatch 2024-03-20T08:43:03.704 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:43:03 smithi183 ceph-mon[25285]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "c1040fce-5e05-4b4f-85b9-da261546233e"}]': finished 2024-03-20T08:43:03.704 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:43:03 smithi183 ceph-mon[25285]: osdmap e15: 3 total, 2 up, 3 in 2024-03-20T08:43:03.705 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:43:03 smithi183 ceph-mon[25285]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-20T08:43:03.705 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:43:03 smithi183 ceph-mon[31860]: from='client.? 172.21.15.183:0/2159720631' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c1040fce-5e05-4b4f-85b9-da261546233e"}]: dispatch 2024-03-20T08:43:03.705 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:43:03 smithi183 ceph-mon[31860]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c1040fce-5e05-4b4f-85b9-da261546233e"}]: dispatch 2024-03-20T08:43:03.705 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:43:03 smithi183 ceph-mon[31860]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "c1040fce-5e05-4b4f-85b9-da261546233e"}]': finished 2024-03-20T08:43:03.705 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:43:03 smithi183 ceph-mon[31860]: osdmap e15: 3 total, 2 up, 3 in 2024-03-20T08:43:03.705 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:43:03 smithi183 ceph-mon[31860]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-20T08:43:04.888 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:43:04 smithi194 ceph-mon[27237]: pgmap v69: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-20T08:43:04.889 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:43:04 smithi194 ceph-mon[27237]: from='client.? 172.21.15.183:0/2253495024' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-20T08:43:04.954 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:43:04 smithi183 ceph-mon[25285]: pgmap v69: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-20T08:43:04.954 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:43:04 smithi183 ceph-mon[25285]: from='client.? 172.21.15.183:0/2253495024' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-20T08:43:04.955 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:43:04 smithi183 ceph-mon[31860]: pgmap v69: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-20T08:43:04.955 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:43:04 smithi183 ceph-mon[31860]: from='client.? 172.21.15.183:0/2253495024' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-20T08:43:06.704 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:43:06 smithi183 ceph-mon[25285]: pgmap v70: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-20T08:43:06.704 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:43:06 smithi183 ceph-mon[25285]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:43:06.705 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:43:06 smithi183 ceph-mon[25285]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:43:06.705 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:43:06 smithi183 ceph-mon[25285]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:43:06.705 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:43:06 smithi183 ceph-mon[31860]: pgmap v70: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-20T08:43:06.705 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:43:06 smithi183 ceph-mon[31860]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:43:06.705 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:43:06 smithi183 ceph-mon[31860]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:43:06.705 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:43:06 smithi183 ceph-mon[31860]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:43:06.890 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:43:06 smithi194 ceph-mon[27237]: pgmap v70: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-20T08:43:06.890 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:43:06 smithi194 ceph-mon[27237]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:43:06.890 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:43:06 smithi194 ceph-mon[27237]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:43:06.890 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:43:06 smithi194 ceph-mon[27237]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:43:08.888 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:43:08 smithi194 ceph-mon[27237]: pgmap v71: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-20T08:43:08.912 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:43:08 smithi183 ceph-mon[25285]: pgmap v71: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-20T08:43:08.913 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:43:08 smithi183 ceph-mon[31860]: pgmap v71: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-20T08:43:09.704 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:43:09 smithi183 ceph-mon[25285]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-03-20T08:43:09.704 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:43:09 smithi183 ceph-mon[25285]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-20T08:43:09.704 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:43:09 smithi183 ceph-mon[31860]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-03-20T08:43:09.704 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:43:09 smithi183 ceph-mon[31860]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-20T08:43:09.888 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:43:09 smithi194 ceph-mon[27237]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-03-20T08:43:09.888 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:43:09 smithi194 ceph-mon[27237]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-20T08:43:10.704 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:43:10 smithi183 ceph-mon[25285]: pgmap v72: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-20T08:43:10.704 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:43:10 smithi183 ceph-mon[25285]: Deploying daemon osd.2 on smithi183 2024-03-20T08:43:10.705 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:43:10 smithi183 ceph-mon[31860]: pgmap v72: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-20T08:43:10.705 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:43:10 smithi183 ceph-mon[31860]: Deploying daemon osd.2 on smithi183 2024-03-20T08:43:10.889 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:43:10 smithi194 ceph-mon[27237]: pgmap v72: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-20T08:43:10.889 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:43:10 smithi194 ceph-mon[27237]: Deploying daemon osd.2 on smithi183 2024-03-20T08:43:12.612 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:43:12 smithi183 ceph-mon[25285]: pgmap v73: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-20T08:43:12.612 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:43:12 smithi183 ceph-mon[31860]: pgmap v73: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-20T08:43:12.889 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:43:12 smithi194 ceph-mon[27237]: pgmap v73: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-20T08:43:14.888 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:43:14 smithi194 ceph-mon[27237]: pgmap v74: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-20T08:43:14.907 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:43:14 smithi183 ceph-mon[25285]: pgmap v74: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-20T08:43:14.907 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:43:14 smithi183 ceph-mon[31860]: pgmap v74: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-20T08:43:16.388 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:43:16 smithi194 ceph-mon[27237]: pgmap v75: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-20T08:43:16.389 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:43:16 smithi194 ceph-mon[27237]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:43:16.389 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:43:16 smithi194 ceph-mon[27237]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:43:16.389 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:43:16 smithi194 ceph-mon[27237]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-20T08:43:16.389 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:43:16 smithi194 ceph-mon[27237]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-20T08:43:16.389 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:43:16 smithi194 ceph-mon[27237]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-20T08:43:16.389 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:43:16 smithi194 ceph-mon[27237]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:43:16.454 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:43:16 smithi183 ceph-mon[31860]: pgmap v75: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-20T08:43:16.454 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:43:16 smithi183 ceph-mon[31860]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:43:16.454 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:43:16 smithi183 ceph-mon[31860]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:43:16.454 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:43:16 smithi183 ceph-mon[31860]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-20T08:43:16.455 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:43:16 smithi183 ceph-mon[31860]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-20T08:43:16.455 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:43:16 smithi183 ceph-mon[31860]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-20T08:43:16.455 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:43:16 smithi183 ceph-mon[31860]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:43:16.455 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:43:16 smithi183 ceph-mon[25285]: pgmap v75: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-20T08:43:16.455 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:43:16 smithi183 ceph-mon[25285]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:43:16.455 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:43:16 smithi183 ceph-mon[25285]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:43:16.455 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:43:16 smithi183 ceph-mon[25285]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-20T08:43:16.455 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:43:16 smithi183 ceph-mon[25285]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-20T08:43:16.455 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:43:16 smithi183 ceph-mon[25285]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-20T08:43:16.455 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:43:16 smithi183 ceph-mon[25285]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:43:18.204 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:43:18 smithi183 ceph-mon[25285]: pgmap v76: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-20T08:43:18.204 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:43:18 smithi183 ceph-mon[31860]: pgmap v76: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-20T08:43:18.388 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:43:18 smithi194 ceph-mon[27237]: pgmap v76: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-20T08:43:19.315 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:43:19 smithi183 ceph-mon[25285]: from='osd.2 [v2:172.21.15.183:6818/1715612189,v1:172.21.15.183:6819/1715612189]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-20T08:43:19.316 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:43:19 smithi183 ceph-mon[31860]: from='osd.2 [v2:172.21.15.183:6818/1715612189,v1:172.21.15.183:6819/1715612189]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-20T08:43:19.388 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:43:19 smithi194 ceph-mon[27237]: from='osd.2 [v2:172.21.15.183:6818/1715612189,v1:172.21.15.183:6819/1715612189]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-20T08:43:20.390 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:43:20 smithi194 ceph-mon[27237]: pgmap v77: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-20T08:43:20.390 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:43:20 smithi194 ceph-mon[27237]: from='osd.2 [v2:172.21.15.183:6818/1715612189,v1:172.21.15.183:6819/1715612189]' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-03-20T08:43:20.390 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:43:20 smithi194 ceph-mon[27237]: osdmap e16: 3 total, 2 up, 3 in 2024-03-20T08:43:20.390 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:43:20 smithi194 ceph-mon[27237]: from='osd.2 [v2:172.21.15.183:6818/1715612189,v1:172.21.15.183:6819/1715612189]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi183", "root=default"]}]: dispatch 2024-03-20T08:43:20.390 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:43:20 smithi194 ceph-mon[27237]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-20T08:43:20.390 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:43:20 smithi194 ceph-mon[27237]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:43:20.390 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:43:20 smithi194 ceph-mon[27237]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:43:20.391 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:43:20 smithi194 ceph-mon[27237]: Detected new or changed devices on smithi183 2024-03-20T08:43:20.391 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:43:20 smithi194 ceph-mon[27237]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:43:20.391 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:43:20 smithi194 ceph-mon[27237]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-20T08:43:20.391 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:43:20 smithi194 ceph-mon[27237]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-20T08:43:20.391 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:43:20 smithi194 ceph-mon[27237]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-20T08:43:20.391 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:43:20 smithi194 ceph-mon[27237]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:43:20.391 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:43:20 smithi183 ceph-mon[31860]: pgmap v77: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-20T08:43:20.391 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:43:20 smithi183 ceph-mon[31860]: from='osd.2 [v2:172.21.15.183:6818/1715612189,v1:172.21.15.183:6819/1715612189]' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-03-20T08:43:20.391 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:43:20 smithi183 ceph-mon[31860]: osdmap e16: 3 total, 2 up, 3 in 2024-03-20T08:43:20.391 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:43:20 smithi183 ceph-mon[31860]: from='osd.2 [v2:172.21.15.183:6818/1715612189,v1:172.21.15.183:6819/1715612189]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi183", "root=default"]}]: dispatch 2024-03-20T08:43:20.391 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:43:20 smithi183 ceph-mon[31860]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-20T08:43:20.391 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:43:20 smithi183 ceph-mon[31860]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:43:20.391 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:43:20 smithi183 ceph-mon[31860]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:43:20.392 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:43:20 smithi183 ceph-mon[31860]: Detected new or changed devices on smithi183 2024-03-20T08:43:20.392 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:43:20 smithi183 ceph-mon[31860]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:43:20.392 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:43:20 smithi183 ceph-mon[31860]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-20T08:43:20.392 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:43:20 smithi183 ceph-mon[31860]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-20T08:43:20.392 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:43:20 smithi183 ceph-mon[31860]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-20T08:43:20.392 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:43:20 smithi183 ceph-mon[31860]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:43:20.392 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:43:20 smithi183 ceph-mon[25285]: pgmap v77: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-20T08:43:20.392 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:43:20 smithi183 ceph-mon[25285]: from='osd.2 [v2:172.21.15.183:6818/1715612189,v1:172.21.15.183:6819/1715612189]' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-03-20T08:43:20.392 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:43:20 smithi183 ceph-mon[25285]: osdmap e16: 3 total, 2 up, 3 in 2024-03-20T08:43:20.392 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:43:20 smithi183 ceph-mon[25285]: from='osd.2 [v2:172.21.15.183:6818/1715612189,v1:172.21.15.183:6819/1715612189]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi183", "root=default"]}]: dispatch 2024-03-20T08:43:20.392 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:43:20 smithi183 ceph-mon[25285]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-20T08:43:20.393 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:43:20 smithi183 ceph-mon[25285]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:43:20.393 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:43:20 smithi183 ceph-mon[25285]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:43:20.393 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:43:20 smithi183 ceph-mon[25285]: Detected new or changed devices on smithi183 2024-03-20T08:43:20.393 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:43:20 smithi183 ceph-mon[25285]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:43:20.393 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:43:20 smithi183 ceph-mon[25285]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-20T08:43:20.393 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:43:20 smithi183 ceph-mon[25285]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-20T08:43:20.393 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:43:20 smithi183 ceph-mon[25285]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-20T08:43:20.393 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:43:20 smithi183 ceph-mon[25285]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:43:20.903 INFO:teuthology.orchestra.run.smithi183.stdout:Created osd(s) 2 on host 'smithi183' 2024-03-20T08:43:21.389 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:43:21 smithi194 ceph-mon[27237]: from='osd.2 [v2:172.21.15.183:6818/1715612189,v1:172.21.15.183:6819/1715612189]' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi183", "root=default"]}]': finished 2024-03-20T08:43:21.389 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:43:21 smithi194 ceph-mon[27237]: osdmap e17: 3 total, 2 up, 3 in 2024-03-20T08:43:21.389 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:43:21 smithi194 ceph-mon[27237]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-20T08:43:21.389 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:43:21 smithi194 ceph-mon[27237]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-20T08:43:21.389 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:43:21 smithi194 ceph-mon[27237]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:43:21.389 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:43:21 smithi194 ceph-mon[27237]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:43:21.389 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:43:21 smithi194 ceph-mon[27237]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-20T08:43:21.389 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:43:21 smithi194 ceph-mon[27237]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-20T08:43:21.389 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:43:21 smithi194 ceph-mon[27237]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-20T08:43:21.389 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:43:21 smithi194 ceph-mon[27237]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:43:21.454 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:43:21 smithi183 ceph-mon[25285]: from='osd.2 [v2:172.21.15.183:6818/1715612189,v1:172.21.15.183:6819/1715612189]' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi183", "root=default"]}]': finished 2024-03-20T08:43:21.454 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:43:21 smithi183 ceph-mon[25285]: osdmap e17: 3 total, 2 up, 3 in 2024-03-20T08:43:21.454 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:43:21 smithi183 ceph-mon[25285]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-20T08:43:21.454 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:43:21 smithi183 ceph-mon[25285]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-20T08:43:21.454 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:43:21 smithi183 ceph-mon[25285]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:43:21.455 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:43:21 smithi183 ceph-mon[25285]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:43:21.455 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:43:21 smithi183 ceph-mon[25285]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-20T08:43:21.455 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:43:21 smithi183 ceph-mon[25285]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-20T08:43:21.455 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:43:21 smithi183 ceph-mon[25285]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-20T08:43:21.455 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:43:21 smithi183 ceph-mon[25285]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:43:21.455 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:43:21 smithi183 ceph-mon[31860]: from='osd.2 [v2:172.21.15.183:6818/1715612189,v1:172.21.15.183:6819/1715612189]' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi183", "root=default"]}]': finished 2024-03-20T08:43:21.455 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:43:21 smithi183 ceph-mon[31860]: osdmap e17: 3 total, 2 up, 3 in 2024-03-20T08:43:21.455 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:43:21 smithi183 ceph-mon[31860]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-20T08:43:21.455 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:43:21 smithi183 ceph-mon[31860]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-20T08:43:21.455 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:43:21 smithi183 ceph-mon[31860]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:43:21.455 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:43:21 smithi183 ceph-mon[31860]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:43:21.456 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:43:21 smithi183 ceph-mon[31860]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-20T08:43:21.456 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:43:21 smithi183 ceph-mon[31860]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-20T08:43:21.456 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:43:21 smithi183 ceph-mon[31860]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-20T08:43:21.456 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:43:21 smithi183 ceph-mon[31860]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:43:21.520 DEBUG:teuthology.orchestra.run.smithi183:osd.2> sudo journalctl -f -n 0 -u ceph-4c3f3eba-e695-11ee-95c9-87774f69a715@osd.2.service 2024-03-20T08:43:21.523 INFO:tasks.cephadm:Deploying osd.3 on smithi183 with /dev/nvme1n1... 2024-03-20T08:43:21.523 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4c3f3eba-e695-11ee-95c9-87774f69a715 -- ceph-volume lvm zap /dev/nvme1n1 2024-03-20T08:43:21.819 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/4c3f3eba-e695-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T08:43:22.389 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:43:22 smithi194 ceph-mon[27237]: purged_snaps scrub starts 2024-03-20T08:43:22.389 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:43:22 smithi194 ceph-mon[27237]: purged_snaps scrub ok 2024-03-20T08:43:22.389 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:43:22 smithi194 ceph-mon[27237]: pgmap v80: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-20T08:43:22.389 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:43:22 smithi194 ceph-mon[27237]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-20T08:43:22.389 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:43:22 smithi194 ceph-mon[27237]: osd.2 [v2:172.21.15.183:6818/1715612189,v1:172.21.15.183:6819/1715612189] boot 2024-03-20T08:43:22.389 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:43:22 smithi194 ceph-mon[27237]: osdmap e18: 3 total, 3 up, 3 in 2024-03-20T08:43:22.389 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:43:22 smithi194 ceph-mon[27237]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-20T08:43:22.389 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:43:22 smithi194 ceph-mon[27237]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-03-20T08:43:22.389 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:43:22 smithi194 ceph-mon[27237]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:43:22.389 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:43:22 smithi194 ceph-mon[27237]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:43:22.389 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:43:22 smithi194 ceph-mon[27237]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:43:22.390 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:43:22 smithi194 ceph-mon[27237]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-20T08:43:22.390 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:43:22 smithi194 ceph-mon[27237]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-20T08:43:22.390 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:43:22 smithi194 ceph-mon[27237]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-20T08:43:22.390 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:43:22 smithi194 ceph-mon[27237]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:43:22.455 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:43:22 smithi183 ceph-mon[25285]: purged_snaps scrub starts 2024-03-20T08:43:22.455 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:43:22 smithi183 ceph-mon[25285]: purged_snaps scrub ok 2024-03-20T08:43:22.455 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:43:22 smithi183 ceph-mon[25285]: pgmap v80: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-20T08:43:22.455 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:43:22 smithi183 ceph-mon[25285]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-20T08:43:22.455 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:43:22 smithi183 ceph-mon[25285]: osd.2 [v2:172.21.15.183:6818/1715612189,v1:172.21.15.183:6819/1715612189] boot 2024-03-20T08:43:22.455 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:43:22 smithi183 ceph-mon[25285]: osdmap e18: 3 total, 3 up, 3 in 2024-03-20T08:43:22.455 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:43:22 smithi183 ceph-mon[25285]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-20T08:43:22.456 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:43:22 smithi183 ceph-mon[25285]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-03-20T08:43:22.456 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:43:22 smithi183 ceph-mon[25285]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:43:22.456 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:43:22 smithi183 ceph-mon[25285]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:43:22.456 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:43:22 smithi183 ceph-mon[25285]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:43:22.456 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:43:22 smithi183 ceph-mon[25285]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-20T08:43:22.456 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:43:22 smithi183 ceph-mon[25285]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-20T08:43:22.456 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:43:22 smithi183 ceph-mon[25285]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-20T08:43:22.456 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:43:22 smithi183 ceph-mon[25285]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:43:22.456 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:43:22 smithi183 ceph-mon[31860]: purged_snaps scrub starts 2024-03-20T08:43:22.456 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:43:22 smithi183 ceph-mon[31860]: purged_snaps scrub ok 2024-03-20T08:43:22.457 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:43:22 smithi183 ceph-mon[31860]: pgmap v80: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-20T08:43:22.457 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:43:22 smithi183 ceph-mon[31860]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-20T08:43:22.457 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:43:22 smithi183 ceph-mon[31860]: osd.2 [v2:172.21.15.183:6818/1715612189,v1:172.21.15.183:6819/1715612189] boot 2024-03-20T08:43:22.457 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:43:22 smithi183 ceph-mon[31860]: osdmap e18: 3 total, 3 up, 3 in 2024-03-20T08:43:22.457 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:43:22 smithi183 ceph-mon[31860]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-20T08:43:22.457 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:43:22 smithi183 ceph-mon[31860]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-03-20T08:43:22.457 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:43:22 smithi183 ceph-mon[31860]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:43:22.457 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:43:22 smithi183 ceph-mon[31860]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:43:22.457 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:43:22 smithi183 ceph-mon[31860]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:43:22.457 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:43:22 smithi183 ceph-mon[31860]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-20T08:43:22.457 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:43:22 smithi183 ceph-mon[31860]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-20T08:43:22.457 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:43:22 smithi183 ceph-mon[31860]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-20T08:43:22.457 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:43:22 smithi183 ceph-mon[31860]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:43:23.204 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:43:23 smithi183 ceph-mon[25285]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-03-20T08:43:23.204 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:43:23 smithi183 ceph-mon[25285]: osdmap e19: 3 total, 3 up, 3 in 2024-03-20T08:43:23.204 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:43:23 smithi183 ceph-mon[25285]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-03-20T08:43:23.205 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:43:23 smithi183 ceph-mon[31860]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-03-20T08:43:23.205 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:43:23 smithi183 ceph-mon[31860]: osdmap e19: 3 total, 3 up, 3 in 2024-03-20T08:43:23.205 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:43:23 smithi183 ceph-mon[31860]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-03-20T08:43:23.388 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:43:23 smithi194 ceph-mon[27237]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-03-20T08:43:23.388 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:43:23 smithi194 ceph-mon[27237]: osdmap e19: 3 total, 3 up, 3 in 2024-03-20T08:43:23.389 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:43:23 smithi194 ceph-mon[27237]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-03-20T08:43:23.515 INFO:journalctl@ceph.osd.0.smithi183.stdout:Mar 20 08:43:23 smithi183 sudo[44790]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme4n1 2024-03-20T08:43:23.515 INFO:journalctl@ceph.osd.0.smithi183.stdout:Mar 20 08:43:23 smithi183 sudo[44790]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-20T08:43:23.515 INFO:journalctl@ceph.osd.0.smithi183.stdout:Mar 20 08:43:23 smithi183 sudo[44790]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-20T08:43:23.515 INFO:journalctl@ceph.osd.0.smithi183.stdout:Mar 20 08:43:23 smithi183 sudo[44790]: pam_unix(sudo:session): session closed for user root 2024-03-20T08:43:23.515 INFO:journalctl@ceph.osd.0.smithi183.stdout:Mar 20 08:43:23 smithi183 sudo[44793]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme linux smart-log-add --json /dev/nvme4n1 2024-03-20T08:43:23.515 INFO:journalctl@ceph.osd.0.smithi183.stdout:Mar 20 08:43:23 smithi183 sudo[44793]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-20T08:43:23.515 INFO:journalctl@ceph.osd.0.smithi183.stdout:Mar 20 08:43:23 smithi183 sudo[44793]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-20T08:43:23.515 INFO:journalctl@ceph.osd.0.smithi183.stdout:Mar 20 08:43:23 smithi183 sudo[44793]: pam_unix(sudo:session): session closed for user root 2024-03-20T08:43:23.515 INFO:journalctl@ceph.osd.1.smithi183.stdout:Mar 20 08:43:23 smithi183 sudo[44796]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme3n1 2024-03-20T08:43:23.515 INFO:journalctl@ceph.osd.1.smithi183.stdout:Mar 20 08:43:23 smithi183 sudo[44796]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-20T08:43:23.516 INFO:journalctl@ceph.osd.1.smithi183.stdout:Mar 20 08:43:23 smithi183 sudo[44796]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-20T08:43:23.516 INFO:journalctl@ceph.osd.1.smithi183.stdout:Mar 20 08:43:23 smithi183 sudo[44796]: pam_unix(sudo:session): session closed for user root 2024-03-20T08:43:23.516 INFO:journalctl@ceph.osd.1.smithi183.stdout:Mar 20 08:43:23 smithi183 sudo[44799]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme linux smart-log-add --json /dev/nvme3n1 2024-03-20T08:43:23.516 INFO:journalctl@ceph.osd.1.smithi183.stdout:Mar 20 08:43:23 smithi183 sudo[44799]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-20T08:43:23.516 INFO:journalctl@ceph.osd.1.smithi183.stdout:Mar 20 08:43:23 smithi183 sudo[44799]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-20T08:43:23.516 INFO:journalctl@ceph.osd.1.smithi183.stdout:Mar 20 08:43:23 smithi183 sudo[44799]: pam_unix(sudo:session): session closed for user root 2024-03-20T08:43:23.867 INFO:teuthology.orchestra.run.smithi183.stderr:--> Zapping: /dev/nvme1n1 2024-03-20T08:43:23.867 INFO:teuthology.orchestra.run.smithi183.stderr:--> --destroy was not specified, but zapping a whole device will remove the partition table 2024-03-20T08:43:23.867 INFO:teuthology.orchestra.run.smithi183.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/nvme1n1 bs=1M count=10 conv=fsync 2024-03-20T08:43:23.867 INFO:teuthology.orchestra.run.smithi183.stderr: stderr: 10+0 records in 2024-03-20T08:43:23.867 INFO:teuthology.orchestra.run.smithi183.stderr:10+0 records out 2024-03-20T08:43:23.867 INFO:teuthology.orchestra.run.smithi183.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0123072 s, 852 MB/s 2024-03-20T08:43:23.867 INFO:teuthology.orchestra.run.smithi183.stderr:--> Zapping successful for: 2024-03-20T08:43:23.868 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:43:23 smithi183 sudo[44811]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-20T08:43:23.868 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:43:23 smithi183 sudo[44811]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-20T08:43:23.868 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:43:23 smithi183 sudo[44811]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-20T08:43:23.868 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:43:23 smithi183 sudo[44811]: pam_unix(sudo:session): session closed for user root 2024-03-20T08:43:23.868 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:43:23 smithi183 sudo[44814]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-20T08:43:23.868 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:43:23 smithi183 sudo[44814]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-20T08:43:23.868 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:43:23 smithi183 sudo[44814]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-20T08:43:23.868 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:43:23 smithi183 sudo[44814]: pam_unix(sudo:session): session closed for user root 2024-03-20T08:43:23.868 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:43:23 smithi183 sudo[44826]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-20T08:43:23.869 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:43:23 smithi183 sudo[44826]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-20T08:43:23.869 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:43:23 smithi183 sudo[44826]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-20T08:43:23.869 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:43:23 smithi183 sudo[44826]: pam_unix(sudo:session): session closed for user root 2024-03-20T08:43:23.869 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:43:23 smithi183 sudo[44832]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-20T08:43:23.869 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:43:23 smithi183 sudo[44832]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-20T08:43:23.869 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:43:23 smithi183 sudo[44832]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-20T08:43:23.869 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:43:23 smithi183 sudo[44832]: pam_unix(sudo:session): session closed for user root 2024-03-20T08:43:23.869 INFO:journalctl@ceph.osd.2.smithi183.stdout:Mar 20 08:43:23 smithi183 sudo[44802]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme2n1 2024-03-20T08:43:23.869 INFO:journalctl@ceph.osd.2.smithi183.stdout:Mar 20 08:43:23 smithi183 sudo[44802]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-20T08:43:23.869 INFO:journalctl@ceph.osd.2.smithi183.stdout:Mar 20 08:43:23 smithi183 sudo[44802]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-20T08:43:23.869 INFO:journalctl@ceph.osd.2.smithi183.stdout:Mar 20 08:43:23 smithi183 sudo[44802]: pam_unix(sudo:session): session closed for user root 2024-03-20T08:43:23.870 INFO:journalctl@ceph.osd.2.smithi183.stdout:Mar 20 08:43:23 smithi183 sudo[44808]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme linux smart-log-add --json /dev/nvme2n1 2024-03-20T08:43:23.870 INFO:journalctl@ceph.osd.2.smithi183.stdout:Mar 20 08:43:23 smithi183 sudo[44808]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-20T08:43:23.870 INFO:journalctl@ceph.osd.2.smithi183.stdout:Mar 20 08:43:23 smithi183 sudo[44808]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-20T08:43:23.870 INFO:journalctl@ceph.osd.2.smithi183.stdout:Mar 20 08:43:23 smithi183 sudo[44808]: pam_unix(sudo:session): session closed for user root 2024-03-20T08:43:23.889 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:43:23 smithi194 sudo[31158]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-20T08:43:23.889 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:43:23 smithi194 sudo[31158]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-20T08:43:23.889 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:43:23 smithi194 sudo[31158]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-20T08:43:23.889 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:43:23 smithi194 sudo[31158]: pam_unix(sudo:session): session closed for user root 2024-03-20T08:43:23.889 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:43:23 smithi194 sudo[31161]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-20T08:43:23.889 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:43:23 smithi194 sudo[31161]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-20T08:43:23.889 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:43:23 smithi194 sudo[31161]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-20T08:43:23.889 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:43:23 smithi194 sudo[31161]: pam_unix(sudo:session): session closed for user root 2024-03-20T08:43:24.389 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:43:24 smithi194 ceph-mon[27237]: pgmap v83: 1 pgs: 1 unknown; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2024-03-20T08:43:24.389 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:43:24 smithi194 ceph-mon[27237]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-20T08:43:24.389 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:43:24 smithi194 ceph-mon[27237]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-03-20T08:43:24.389 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:43:24 smithi194 ceph-mon[27237]: osdmap e20: 3 total, 3 up, 3 in 2024-03-20T08:43:24.389 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:43:24 smithi194 ceph-mon[27237]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:43:24.389 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:43:24 smithi194 ceph-mon[27237]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:43:24.390 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:43:24 smithi194 ceph-mon[27237]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:43:24.390 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:43:24 smithi194 ceph-mon[27237]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-20T08:43:24.390 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:43:24 smithi194 ceph-mon[27237]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-20T08:43:24.390 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:43:24 smithi194 ceph-mon[27237]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-20T08:43:24.390 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:43:24 smithi194 ceph-mon[27237]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-20T08:43:24.390 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:43:24 smithi194 ceph-mon[27237]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-20T08:43:24.390 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:43:24 smithi194 ceph-mon[27237]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:43:24.390 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:43:24 smithi194 ceph-mon[27237]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-20T08:43:24.390 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:43:24 smithi194 ceph-mon[27237]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-20T08:43:24.391 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:43:24 smithi194 ceph-mon[27237]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-20T08:43:24.391 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:43:24 smithi194 ceph-mon[27237]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-20T08:43:24.391 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:43:24 smithi194 ceph-mon[27237]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-20T08:43:24.391 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:43:24 smithi194 ceph-mon[27237]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:43:24.391 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:43:24 smithi194 ceph-mon[27237]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-20T08:43:24.391 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:43:24 smithi194 ceph-mon[27237]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-20T08:43:24.391 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:43:24 smithi194 ceph-mon[27237]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-20T08:43:24.391 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:43:24 smithi194 ceph-mon[27237]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-20T08:43:24.391 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:43:24 smithi194 ceph-mon[27237]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-20T08:43:24.455 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:43:24 smithi183 ceph-mon[25285]: pgmap v83: 1 pgs: 1 unknown; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2024-03-20T08:43:24.455 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:43:24 smithi183 ceph-mon[25285]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-20T08:43:24.455 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:43:24 smithi183 ceph-mon[25285]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-03-20T08:43:24.455 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:43:24 smithi183 ceph-mon[25285]: osdmap e20: 3 total, 3 up, 3 in 2024-03-20T08:43:24.455 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:43:24 smithi183 ceph-mon[25285]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:43:24.455 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:43:24 smithi183 ceph-mon[25285]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:43:24.455 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:43:24 smithi183 ceph-mon[25285]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:43:24.455 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:43:24 smithi183 ceph-mon[25285]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-20T08:43:24.455 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:43:24 smithi183 ceph-mon[25285]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-20T08:43:24.455 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:43:24 smithi183 ceph-mon[25285]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-20T08:43:24.456 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:43:24 smithi183 ceph-mon[25285]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-20T08:43:24.456 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:43:24 smithi183 ceph-mon[25285]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-20T08:43:24.456 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:43:24 smithi183 ceph-mon[25285]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:43:24.456 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:43:24 smithi183 ceph-mon[25285]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-20T08:43:24.456 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:43:24 smithi183 ceph-mon[25285]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-20T08:43:24.456 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:43:24 smithi183 ceph-mon[25285]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-20T08:43:24.456 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:43:24 smithi183 ceph-mon[25285]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-20T08:43:24.456 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:43:24 smithi183 ceph-mon[25285]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-20T08:43:24.456 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:43:24 smithi183 ceph-mon[25285]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:43:24.456 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:43:24 smithi183 ceph-mon[25285]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-20T08:43:24.456 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:43:24 smithi183 ceph-mon[25285]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-20T08:43:24.457 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:43:24 smithi183 ceph-mon[25285]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-20T08:43:24.457 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:43:24 smithi183 ceph-mon[25285]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-20T08:43:24.457 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:43:24 smithi183 ceph-mon[25285]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-20T08:43:24.457 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:43:24 smithi183 ceph-mon[31860]: pgmap v83: 1 pgs: 1 unknown; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2024-03-20T08:43:24.457 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:43:24 smithi183 ceph-mon[31860]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-20T08:43:24.457 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:43:24 smithi183 ceph-mon[31860]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-03-20T08:43:24.457 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:43:24 smithi183 ceph-mon[31860]: osdmap e20: 3 total, 3 up, 3 in 2024-03-20T08:43:24.457 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:43:24 smithi183 ceph-mon[31860]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:43:24.457 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:43:24 smithi183 ceph-mon[31860]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:43:24.458 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:43:24 smithi183 ceph-mon[31860]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:43:24.458 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:43:24 smithi183 ceph-mon[31860]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-20T08:43:24.458 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:43:24 smithi183 ceph-mon[31860]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-20T08:43:24.458 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:43:24 smithi183 ceph-mon[31860]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-20T08:43:24.458 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:43:24 smithi183 ceph-mon[31860]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-20T08:43:24.458 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:43:24 smithi183 ceph-mon[31860]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-20T08:43:24.458 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:43:24 smithi183 ceph-mon[31860]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:43:24.458 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:43:24 smithi183 ceph-mon[31860]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-20T08:43:24.458 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:43:24 smithi183 ceph-mon[31860]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-20T08:43:24.459 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:43:24 smithi183 ceph-mon[31860]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-20T08:43:24.459 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:43:24 smithi183 ceph-mon[31860]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-20T08:43:24.459 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:43:24 smithi183 ceph-mon[31860]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-20T08:43:24.459 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:43:24 smithi183 ceph-mon[31860]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:43:24.459 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:43:24 smithi183 ceph-mon[31860]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-20T08:43:24.459 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:43:24 smithi183 ceph-mon[31860]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-20T08:43:24.459 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:43:24 smithi183 ceph-mon[31860]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-20T08:43:24.459 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:43:24 smithi183 ceph-mon[31860]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-20T08:43:24.459 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:43:24 smithi183 ceph-mon[31860]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-20T08:43:24.487 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4c3f3eba-e695-11ee-95c9-87774f69a715 -- ceph orch daemon add osd smithi183:/dev/nvme1n1 2024-03-20T08:43:24.725 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/4c3f3eba-e695-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T08:43:25.388 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:43:25 smithi194 ceph-mon[27237]: osdmap e21: 3 total, 3 up, 3 in 2024-03-20T08:43:25.454 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:43:25 smithi183 ceph-mon[25285]: osdmap e21: 3 total, 3 up, 3 in 2024-03-20T08:43:25.454 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:43:25 smithi183 ceph-mon[31860]: osdmap e21: 3 total, 3 up, 3 in 2024-03-20T08:43:26.388 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:43:26 smithi194 ceph-mon[27237]: pgmap v86: 1 pgs: 1 unknown; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2024-03-20T08:43:26.388 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:43:26 smithi194 ceph-mon[27237]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-03-20T08:43:26.389 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:43:26 smithi194 ceph-mon[27237]: Cluster is now healthy 2024-03-20T08:43:26.389 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:43:26 smithi194 ceph-mon[27237]: mgrmap e15: y(active, since 2m), standbys: x 2024-03-20T08:43:26.454 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:43:26 smithi183 ceph-mon[31860]: pgmap v86: 1 pgs: 1 unknown; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2024-03-20T08:43:26.454 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:43:26 smithi183 ceph-mon[31860]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-03-20T08:43:26.454 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:43:26 smithi183 ceph-mon[31860]: Cluster is now healthy 2024-03-20T08:43:26.454 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:43:26 smithi183 ceph-mon[31860]: mgrmap e15: y(active, since 2m), standbys: x 2024-03-20T08:43:26.454 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:43:26 smithi183 ceph-mon[25285]: pgmap v86: 1 pgs: 1 unknown; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2024-03-20T08:43:26.455 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:43:26 smithi183 ceph-mon[25285]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-03-20T08:43:26.455 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:43:26 smithi183 ceph-mon[25285]: Cluster is now healthy 2024-03-20T08:43:26.455 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:43:26 smithi183 ceph-mon[25285]: mgrmap e15: y(active, since 2m), standbys: x 2024-03-20T08:43:27.545 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:43:27 smithi194 ceph-mon[27237]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:43:27.545 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:43:27 smithi194 ceph-mon[27237]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:43:27.545 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:43:27 smithi194 ceph-mon[27237]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:43:27.546 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:43:27 smithi194 ceph-mon[27237]: from='client.24209 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi183:/dev/nvme1n1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-20T08:43:27.546 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:43:27 smithi194 ceph-mon[27237]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-20T08:43:27.546 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:43:27 smithi194 ceph-mon[27237]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-20T08:43:27.546 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:43:27 smithi194 ceph-mon[27237]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-20T08:43:27.704 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:43:27 smithi183 ceph-mon[31860]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:43:27.704 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:43:27 smithi183 ceph-mon[31860]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:43:27.704 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:43:27 smithi183 ceph-mon[31860]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:43:27.705 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:43:27 smithi183 ceph-mon[31860]: from='client.24209 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi183:/dev/nvme1n1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-20T08:43:27.705 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:43:27 smithi183 ceph-mon[31860]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-20T08:43:27.705 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:43:27 smithi183 ceph-mon[31860]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-20T08:43:27.705 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:43:27 smithi183 ceph-mon[31860]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-20T08:43:27.705 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:43:27 smithi183 ceph-mon[25285]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:43:27.705 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:43:27 smithi183 ceph-mon[25285]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:43:27.705 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:43:27 smithi183 ceph-mon[25285]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:43:27.705 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:43:27 smithi183 ceph-mon[25285]: from='client.24209 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi183:/dev/nvme1n1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-20T08:43:27.705 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:43:27 smithi183 ceph-mon[25285]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-20T08:43:27.705 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:43:27 smithi183 ceph-mon[25285]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-20T08:43:27.705 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:43:27 smithi183 ceph-mon[25285]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-20T08:43:28.745 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:43:28 smithi194 ceph-mon[27237]: pgmap v87: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail 2024-03-20T08:43:28.959 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:43:28 smithi183 ceph-mon[25285]: pgmap v87: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail 2024-03-20T08:43:28.960 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:43:28 smithi183 ceph-mon[31860]: pgmap v87: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail 2024-03-20T08:43:29.638 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:43:29 smithi194 ceph-mon[27237]: from='client.? 172.21.15.183:0/3541219775' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c47d2b57-e03c-48ce-872a-5458833fc3d1"}]: dispatch 2024-03-20T08:43:29.638 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:43:29 smithi194 ceph-mon[27237]: from='client.? 172.21.15.183:0/3541219775' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "c47d2b57-e03c-48ce-872a-5458833fc3d1"}]': finished 2024-03-20T08:43:29.639 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:43:29 smithi194 ceph-mon[27237]: osdmap e22: 4 total, 3 up, 4 in 2024-03-20T08:43:29.639 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:43:29 smithi194 ceph-mon[27237]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-20T08:43:29.954 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:43:29 smithi183 ceph-mon[25285]: from='client.? 172.21.15.183:0/3541219775' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c47d2b57-e03c-48ce-872a-5458833fc3d1"}]: dispatch 2024-03-20T08:43:29.954 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:43:29 smithi183 ceph-mon[25285]: from='client.? 172.21.15.183:0/3541219775' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "c47d2b57-e03c-48ce-872a-5458833fc3d1"}]': finished 2024-03-20T08:43:29.955 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:43:29 smithi183 ceph-mon[25285]: osdmap e22: 4 total, 3 up, 4 in 2024-03-20T08:43:29.955 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:43:29 smithi183 ceph-mon[25285]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-20T08:43:29.955 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:43:29 smithi183 ceph-mon[31860]: from='client.? 172.21.15.183:0/3541219775' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c47d2b57-e03c-48ce-872a-5458833fc3d1"}]: dispatch 2024-03-20T08:43:29.955 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:43:29 smithi183 ceph-mon[31860]: from='client.? 172.21.15.183:0/3541219775' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "c47d2b57-e03c-48ce-872a-5458833fc3d1"}]': finished 2024-03-20T08:43:29.955 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:43:29 smithi183 ceph-mon[31860]: osdmap e22: 4 total, 3 up, 4 in 2024-03-20T08:43:29.955 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:43:29 smithi183 ceph-mon[31860]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-20T08:43:30.888 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:43:30 smithi194 ceph-mon[27237]: pgmap v88: 1 pgs: 1 active+clean; 577 KiB data, 87 MiB used, 268 GiB / 268 GiB avail 2024-03-20T08:43:30.955 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:43:30 smithi183 ceph-mon[31860]: pgmap v88: 1 pgs: 1 active+clean; 577 KiB data, 87 MiB used, 268 GiB / 268 GiB avail 2024-03-20T08:43:30.955 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:43:30 smithi183 ceph-mon[25285]: pgmap v88: 1 pgs: 1 active+clean; 577 KiB data, 87 MiB used, 268 GiB / 268 GiB avail 2024-03-20T08:43:31.888 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:43:31 smithi194 ceph-mon[27237]: from='client.? 172.21.15.183:0/489862680' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-20T08:43:31.954 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:43:31 smithi183 ceph-mon[25285]: from='client.? 172.21.15.183:0/489862680' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-20T08:43:31.954 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:43:31 smithi183 ceph-mon[31860]: from='client.? 172.21.15.183:0/489862680' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-20T08:43:32.840 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:43:32 smithi183 ceph-mon[25285]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 87 MiB used, 268 GiB / 268 GiB avail 2024-03-20T08:43:32.840 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:43:32 smithi183 ceph-mon[31860]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 87 MiB used, 268 GiB / 268 GiB avail 2024-03-20T08:43:32.888 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:43:32 smithi194 ceph-mon[27237]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 87 MiB used, 268 GiB / 268 GiB avail 2024-03-20T08:43:34.704 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:43:34 smithi183 ceph-mon[25285]: pgmap v91: 1 pgs: 1 active+clean; 577 KiB data, 85 MiB used, 268 GiB / 268 GiB avail 2024-03-20T08:43:34.705 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:43:34 smithi183 ceph-mon[31860]: pgmap v91: 1 pgs: 1 active+clean; 577 KiB data, 85 MiB used, 268 GiB / 268 GiB avail 2024-03-20T08:43:34.888 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:43:34 smithi194 ceph-mon[27237]: pgmap v91: 1 pgs: 1 active+clean; 577 KiB data, 85 MiB used, 268 GiB / 268 GiB avail 2024-03-20T08:43:36.704 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:43:36 smithi183 ceph-mon[25285]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 85 MiB used, 268 GiB / 268 GiB avail 2024-03-20T08:43:36.705 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:43:36 smithi183 ceph-mon[31860]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 85 MiB used, 268 GiB / 268 GiB avail 2024-03-20T08:43:36.888 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:43:36 smithi194 ceph-mon[27237]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 85 MiB used, 268 GiB / 268 GiB avail 2024-03-20T08:43:38.846 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:43:38 smithi194 ceph-mon[27237]: pgmap v93: 1 pgs: 1 active+clean; 577 KiB data, 85 MiB used, 268 GiB / 268 GiB avail 2024-03-20T08:43:38.846 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:43:38 smithi194 ceph-mon[27237]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-03-20T08:43:38.846 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:43:38 smithi194 ceph-mon[27237]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-20T08:43:38.846 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:43:38 smithi194 ceph-mon[27237]: Deploying daemon osd.3 on smithi183 2024-03-20T08:43:38.846 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:43:38 smithi194 ceph-mon[27237]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:43:38.846 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:43:38 smithi194 ceph-mon[27237]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:43:38.846 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:43:38 smithi194 ceph-mon[27237]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:43:38.955 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:43:38 smithi183 ceph-mon[31860]: pgmap v93: 1 pgs: 1 active+clean; 577 KiB data, 85 MiB used, 268 GiB / 268 GiB avail 2024-03-20T08:43:38.955 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:43:38 smithi183 ceph-mon[31860]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-03-20T08:43:38.955 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:43:38 smithi183 ceph-mon[31860]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-20T08:43:38.955 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:43:38 smithi183 ceph-mon[31860]: Deploying daemon osd.3 on smithi183 2024-03-20T08:43:38.955 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:43:38 smithi183 ceph-mon[31860]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:43:38.955 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:43:38 smithi183 ceph-mon[31860]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:43:38.955 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:43:38 smithi183 ceph-mon[31860]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:43:38.955 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:43:38 smithi183 ceph-mon[25285]: pgmap v93: 1 pgs: 1 active+clean; 577 KiB data, 85 MiB used, 268 GiB / 268 GiB avail 2024-03-20T08:43:38.955 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:43:38 smithi183 ceph-mon[25285]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-03-20T08:43:38.956 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:43:38 smithi183 ceph-mon[25285]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-20T08:43:38.956 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:43:38 smithi183 ceph-mon[25285]: Deploying daemon osd.3 on smithi183 2024-03-20T08:43:38.956 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:43:38 smithi183 ceph-mon[25285]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:43:38.956 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:43:38 smithi183 ceph-mon[25285]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:43:38.956 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:43:38 smithi183 ceph-mon[25285]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:43:40.528 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:43:40 smithi183 ceph-mon[25285]: pgmap v94: 1 pgs: 1 active+clean; 577 KiB data, 85 MiB used, 268 GiB / 268 GiB avail 2024-03-20T08:43:40.528 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:43:40 smithi183 ceph-mon[31860]: pgmap v94: 1 pgs: 1 active+clean; 577 KiB data, 85 MiB used, 268 GiB / 268 GiB avail 2024-03-20T08:43:40.790 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:43:40 smithi194 ceph-mon[27237]: pgmap v94: 1 pgs: 1 active+clean; 577 KiB data, 85 MiB used, 268 GiB / 268 GiB avail 2024-03-20T08:43:42.747 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:43:42 smithi183 ceph-mon[25285]: pgmap v95: 1 pgs: 1 active+clean; 577 KiB data, 85 MiB used, 268 GiB / 268 GiB avail 2024-03-20T08:43:42.748 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:43:42 smithi183 ceph-mon[31860]: pgmap v95: 1 pgs: 1 active+clean; 577 KiB data, 85 MiB used, 268 GiB / 268 GiB avail 2024-03-20T08:43:42.888 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:43:42 smithi194 ceph-mon[27237]: pgmap v95: 1 pgs: 1 active+clean; 577 KiB data, 85 MiB used, 268 GiB / 268 GiB avail 2024-03-20T08:43:44.639 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:43:44 smithi194 ceph-mon[27237]: pgmap v96: 1 pgs: 1 active+clean; 577 KiB data, 85 MiB used, 268 GiB / 268 GiB avail 2024-03-20T08:43:44.662 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:43:44 smithi183 ceph-mon[25285]: pgmap v96: 1 pgs: 1 active+clean; 577 KiB data, 85 MiB used, 268 GiB / 268 GiB avail 2024-03-20T08:43:44.662 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:43:44 smithi183 ceph-mon[31860]: pgmap v96: 1 pgs: 1 active+clean; 577 KiB data, 85 MiB used, 268 GiB / 268 GiB avail 2024-03-20T08:43:45.984 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:43:45 smithi183 ceph-mon[31860]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:43:45.984 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:43:45 smithi183 ceph-mon[31860]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:43:45.984 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:43:45 smithi183 ceph-mon[31860]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-20T08:43:45.985 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:43:45 smithi183 ceph-mon[31860]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-20T08:43:45.985 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:43:45 smithi183 ceph-mon[31860]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-20T08:43:45.985 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:43:45 smithi183 ceph-mon[31860]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:43:45.985 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:43:45 smithi183 ceph-mon[31860]: pgmap v97: 1 pgs: 1 active+clean; 577 KiB data, 85 MiB used, 268 GiB / 268 GiB avail 2024-03-20T08:43:45.985 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:43:45 smithi183 ceph-mon[25285]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:43:45.985 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:43:45 smithi183 ceph-mon[25285]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:43:45.985 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:43:45 smithi183 ceph-mon[25285]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-20T08:43:45.985 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:43:45 smithi183 ceph-mon[25285]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-20T08:43:45.985 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:43:45 smithi183 ceph-mon[25285]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-20T08:43:45.985 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:43:45 smithi183 ceph-mon[25285]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:43:45.985 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:43:45 smithi183 ceph-mon[25285]: pgmap v97: 1 pgs: 1 active+clean; 577 KiB data, 85 MiB used, 268 GiB / 268 GiB avail 2024-03-20T08:43:46.139 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:43:45 smithi194 ceph-mon[27237]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:43:46.139 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:43:45 smithi194 ceph-mon[27237]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:43:46.139 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:43:45 smithi194 ceph-mon[27237]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-20T08:43:46.139 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:43:45 smithi194 ceph-mon[27237]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-20T08:43:46.139 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:43:45 smithi194 ceph-mon[27237]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-20T08:43:46.140 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:43:45 smithi194 ceph-mon[27237]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:43:46.140 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:43:45 smithi194 ceph-mon[27237]: pgmap v97: 1 pgs: 1 active+clean; 577 KiB data, 85 MiB used, 268 GiB / 268 GiB avail 2024-03-20T08:43:47.824 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:43:47 smithi194 ceph-mon[27237]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:43:47.824 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:43:47 smithi194 ceph-mon[27237]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:43:47.825 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:43:47 smithi194 ceph-mon[27237]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:43:47.875 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:43:47 smithi183 ceph-mon[25285]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:43:47.875 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:43:47 smithi183 ceph-mon[25285]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:43:47.875 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:43:47 smithi183 ceph-mon[25285]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:43:47.876 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:43:47 smithi183 ceph-mon[31860]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:43:47.876 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:43:47 smithi183 ceph-mon[31860]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:43:47.876 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:43:47 smithi183 ceph-mon[31860]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:43:47.923 INFO:teuthology.orchestra.run.smithi183.stdout:Created osd(s) 3 on host 'smithi183' 2024-03-20T08:43:48.679 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:43:48 smithi183 ceph-mon[25285]: pgmap v98: 1 pgs: 1 active+clean; 577 KiB data, 85 MiB used, 268 GiB / 268 GiB avail 2024-03-20T08:43:48.679 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:43:48 smithi183 ceph-mon[25285]: from='osd.3 [v2:172.21.15.183:6826/4169086406,v1:172.21.15.183:6827/4169086406]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-03-20T08:43:48.679 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:43:48 smithi183 ceph-mon[25285]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:43:48.679 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:43:48 smithi183 ceph-mon[25285]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:43:48.679 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:43:48 smithi183 ceph-mon[25285]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-20T08:43:48.679 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:43:48 smithi183 ceph-mon[25285]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-20T08:43:48.679 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:43:48 smithi183 ceph-mon[25285]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-20T08:43:48.679 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:43:48 smithi183 ceph-mon[25285]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-20T08:43:48.679 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:43:48 smithi183 ceph-mon[25285]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-20T08:43:48.680 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:43:48 smithi183 ceph-mon[25285]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:43:48.680 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:43:48 smithi183 ceph-mon[25285]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-20T08:43:48.680 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:43:48 smithi183 ceph-mon[25285]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-20T08:43:48.680 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:43:48 smithi183 ceph-mon[25285]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:43:48.680 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:43:48 smithi183 ceph-mon[31860]: pgmap v98: 1 pgs: 1 active+clean; 577 KiB data, 85 MiB used, 268 GiB / 268 GiB avail 2024-03-20T08:43:48.680 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:43:48 smithi183 ceph-mon[31860]: from='osd.3 [v2:172.21.15.183:6826/4169086406,v1:172.21.15.183:6827/4169086406]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-03-20T08:43:48.680 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:43:48 smithi183 ceph-mon[31860]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:43:48.680 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:43:48 smithi183 ceph-mon[31860]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:43:48.680 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:43:48 smithi183 ceph-mon[31860]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-20T08:43:48.680 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:43:48 smithi183 ceph-mon[31860]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-20T08:43:48.680 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:43:48 smithi183 ceph-mon[31860]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-20T08:43:48.681 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:43:48 smithi183 ceph-mon[31860]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-20T08:43:48.681 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:43:48 smithi183 ceph-mon[31860]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-20T08:43:48.681 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:43:48 smithi183 ceph-mon[31860]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:43:48.681 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:43:48 smithi183 ceph-mon[31860]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-20T08:43:48.681 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:43:48 smithi183 ceph-mon[31860]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-20T08:43:48.681 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:43:48 smithi183 ceph-mon[31860]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:43:48.705 DEBUG:teuthology.orchestra.run.smithi183:osd.3> sudo journalctl -f -n 0 -u ceph-4c3f3eba-e695-11ee-95c9-87774f69a715@osd.3.service 2024-03-20T08:43:48.708 INFO:tasks.cephadm:Deploying osd.4 on smithi194 with /dev/nvme4n1... 2024-03-20T08:43:48.708 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4c3f3eba-e695-11ee-95c9-87774f69a715 -- ceph-volume lvm zap /dev/nvme4n1 2024-03-20T08:43:48.889 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:43:48 smithi194 ceph-mon[27237]: pgmap v98: 1 pgs: 1 active+clean; 577 KiB data, 85 MiB used, 268 GiB / 268 GiB avail 2024-03-20T08:43:48.889 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:43:48 smithi194 ceph-mon[27237]: from='osd.3 [v2:172.21.15.183:6826/4169086406,v1:172.21.15.183:6827/4169086406]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-03-20T08:43:48.889 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:43:48 smithi194 ceph-mon[27237]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:43:48.889 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:43:48 smithi194 ceph-mon[27237]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:43:48.889 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:43:48 smithi194 ceph-mon[27237]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-20T08:43:48.889 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:43:48 smithi194 ceph-mon[27237]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-20T08:43:48.889 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:43:48 smithi194 ceph-mon[27237]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-20T08:43:48.889 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:43:48 smithi194 ceph-mon[27237]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-20T08:43:48.889 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:43:48 smithi194 ceph-mon[27237]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-20T08:43:48.889 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:43:48 smithi194 ceph-mon[27237]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:43:48.890 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:43:48 smithi194 ceph-mon[27237]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-20T08:43:48.890 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:43:48 smithi194 ceph-mon[27237]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-20T08:43:48.890 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:43:48 smithi194 ceph-mon[27237]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:43:49.486 INFO:teuthology.orchestra.run.smithi194.stderr:Inferring config /var/lib/ceph/4c3f3eba-e695-11ee-95c9-87774f69a715/mon.b/config 2024-03-20T08:43:49.866 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:43:49 smithi194 ceph-mon[27237]: Adjusting osd_memory_target on smithi183 to 3760M 2024-03-20T08:43:49.866 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:43:49 smithi194 ceph-mon[27237]: from='osd.3 [v2:172.21.15.183:6826/4169086406,v1:172.21.15.183:6827/4169086406]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-03-20T08:43:49.866 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:43:49 smithi194 ceph-mon[27237]: osdmap e23: 4 total, 3 up, 4 in 2024-03-20T08:43:49.867 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:43:49 smithi194 ceph-mon[27237]: from='osd.3 [v2:172.21.15.183:6826/4169086406,v1:172.21.15.183:6827/4169086406]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi183", "root=default"]}]: dispatch 2024-03-20T08:43:49.867 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:43:49 smithi194 ceph-mon[27237]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-20T08:43:49.867 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:43:49 smithi194 ceph-mon[27237]: from='osd.3 [v2:172.21.15.183:6826/4169086406,v1:172.21.15.183:6827/4169086406]' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi183", "root=default"]}]': finished 2024-03-20T08:43:49.867 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:43:49 smithi194 ceph-mon[27237]: osdmap e24: 4 total, 3 up, 4 in 2024-03-20T08:43:50.063 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:43:49 smithi183 ceph-mon[25285]: Adjusting osd_memory_target on smithi183 to 3760M 2024-03-20T08:43:50.063 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:43:49 smithi183 ceph-mon[25285]: from='osd.3 [v2:172.21.15.183:6826/4169086406,v1:172.21.15.183:6827/4169086406]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-03-20T08:43:50.063 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:43:49 smithi183 ceph-mon[25285]: osdmap e23: 4 total, 3 up, 4 in 2024-03-20T08:43:50.063 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:43:49 smithi183 ceph-mon[25285]: from='osd.3 [v2:172.21.15.183:6826/4169086406,v1:172.21.15.183:6827/4169086406]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi183", "root=default"]}]: dispatch 2024-03-20T08:43:50.063 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:43:49 smithi183 ceph-mon[25285]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-20T08:43:50.063 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:43:49 smithi183 ceph-mon[25285]: from='osd.3 [v2:172.21.15.183:6826/4169086406,v1:172.21.15.183:6827/4169086406]' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi183", "root=default"]}]': finished 2024-03-20T08:43:50.063 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:43:49 smithi183 ceph-mon[25285]: osdmap e24: 4 total, 3 up, 4 in 2024-03-20T08:43:50.064 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:43:49 smithi183 ceph-mon[31860]: Adjusting osd_memory_target on smithi183 to 3760M 2024-03-20T08:43:50.064 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:43:49 smithi183 ceph-mon[31860]: from='osd.3 [v2:172.21.15.183:6826/4169086406,v1:172.21.15.183:6827/4169086406]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-03-20T08:43:50.064 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:43:49 smithi183 ceph-mon[31860]: osdmap e23: 4 total, 3 up, 4 in 2024-03-20T08:43:50.064 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:43:49 smithi183 ceph-mon[31860]: from='osd.3 [v2:172.21.15.183:6826/4169086406,v1:172.21.15.183:6827/4169086406]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi183", "root=default"]}]: dispatch 2024-03-20T08:43:50.064 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:43:49 smithi183 ceph-mon[31860]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-20T08:43:50.064 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:43:49 smithi183 ceph-mon[31860]: from='osd.3 [v2:172.21.15.183:6826/4169086406,v1:172.21.15.183:6827/4169086406]' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi183", "root=default"]}]': finished 2024-03-20T08:43:50.064 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:43:49 smithi183 ceph-mon[31860]: osdmap e24: 4 total, 3 up, 4 in 2024-03-20T08:43:50.454 INFO:journalctl@ceph.osd.3.smithi183.stdout:Mar 20 08:43:50 smithi183 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-osd-3[47180]: 2024-03-20T08:43:50.062+0000 7f54380b0700 -1 osd.3 0 waiting for initial osdmap 2024-03-20T08:43:50.455 INFO:journalctl@ceph.osd.3.smithi183.stdout:Mar 20 08:43:50 smithi183 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-osd-3[47180]: 2024-03-20T08:43:50.067+0000 7f542f69a700 -1 osd.3 24 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-20T08:43:51.038 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:43:50 smithi194 ceph-mon[27237]: pgmap v100: 1 pgs: 1 active+clean; 577 KiB data, 85 MiB used, 268 GiB / 268 GiB avail 2024-03-20T08:43:51.038 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:43:50 smithi194 ceph-mon[27237]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-20T08:43:51.038 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:43:50 smithi194 ceph-mon[27237]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-20T08:43:51.039 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:43:50 smithi194 ceph-mon[27237]: from='osd.3 [v2:172.21.15.183:6826/4169086406,v1:172.21.15.183:6827/4169086406]' entity='osd.3' 2024-03-20T08:43:51.039 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:43:50 smithi194 ceph-mon[27237]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-20T08:43:51.204 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:43:50 smithi183 ceph-mon[31860]: pgmap v100: 1 pgs: 1 active+clean; 577 KiB data, 85 MiB used, 268 GiB / 268 GiB avail 2024-03-20T08:43:51.204 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:43:50 smithi183 ceph-mon[31860]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-20T08:43:51.204 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:43:50 smithi183 ceph-mon[31860]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-20T08:43:51.204 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:43:50 smithi183 ceph-mon[31860]: from='osd.3 [v2:172.21.15.183:6826/4169086406,v1:172.21.15.183:6827/4169086406]' entity='osd.3' 2024-03-20T08:43:51.205 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:43:50 smithi183 ceph-mon[31860]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-20T08:43:51.205 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:43:50 smithi183 ceph-mon[25285]: pgmap v100: 1 pgs: 1 active+clean; 577 KiB data, 85 MiB used, 268 GiB / 268 GiB avail 2024-03-20T08:43:51.205 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:43:50 smithi183 ceph-mon[25285]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-20T08:43:51.205 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:43:50 smithi183 ceph-mon[25285]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-20T08:43:51.205 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:43:50 smithi183 ceph-mon[25285]: from='osd.3 [v2:172.21.15.183:6826/4169086406,v1:172.21.15.183:6827/4169086406]' entity='osd.3' 2024-03-20T08:43:51.205 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:43:50 smithi183 ceph-mon[25285]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-20T08:43:51.384 INFO:teuthology.orchestra.run.smithi194.stderr:--> Zapping: /dev/nvme4n1 2024-03-20T08:43:51.384 INFO:teuthology.orchestra.run.smithi194.stderr:--> --destroy was not specified, but zapping a whole device will remove the partition table 2024-03-20T08:43:51.385 INFO:teuthology.orchestra.run.smithi194.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/nvme4n1 bs=1M count=10 conv=fsync 2024-03-20T08:43:51.385 INFO:teuthology.orchestra.run.smithi194.stderr: stderr: 10+0 records in 2024-03-20T08:43:51.385 INFO:teuthology.orchestra.run.smithi194.stderr:10+0 records out 2024-03-20T08:43:51.385 INFO:teuthology.orchestra.run.smithi194.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0143027 s, 733 MB/s 2024-03-20T08:43:51.385 INFO:teuthology.orchestra.run.smithi194.stderr:--> Zapping successful for: 2024-03-20T08:43:52.001 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:43:51 smithi194 ceph-mon[27237]: purged_snaps scrub starts 2024-03-20T08:43:52.001 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:43:51 smithi194 ceph-mon[27237]: purged_snaps scrub ok 2024-03-20T08:43:52.001 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:43:51 smithi194 ceph-mon[27237]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 85 MiB used, 268 GiB / 268 GiB avail 2024-03-20T08:43:52.001 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:43:51 smithi194 ceph-mon[27237]: osd.3 [v2:172.21.15.183:6826/4169086406,v1:172.21.15.183:6827/4169086406] boot 2024-03-20T08:43:52.001 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:43:51 smithi194 ceph-mon[27237]: osdmap e25: 4 total, 4 up, 4 in 2024-03-20T08:43:52.001 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:43:51 smithi194 ceph-mon[27237]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-20T08:43:52.032 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4c3f3eba-e695-11ee-95c9-87774f69a715 -- ceph orch daemon add osd smithi194:/dev/nvme4n1 2024-03-20T08:43:52.204 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:43:51 smithi183 ceph-mon[25285]: purged_snaps scrub starts 2024-03-20T08:43:52.204 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:43:51 smithi183 ceph-mon[25285]: purged_snaps scrub ok 2024-03-20T08:43:52.204 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:43:51 smithi183 ceph-mon[25285]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 85 MiB used, 268 GiB / 268 GiB avail 2024-03-20T08:43:52.204 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:43:51 smithi183 ceph-mon[25285]: osd.3 [v2:172.21.15.183:6826/4169086406,v1:172.21.15.183:6827/4169086406] boot 2024-03-20T08:43:52.205 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:43:51 smithi183 ceph-mon[25285]: osdmap e25: 4 total, 4 up, 4 in 2024-03-20T08:43:52.205 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:43:51 smithi183 ceph-mon[25285]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-20T08:43:52.205 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:43:51 smithi183 ceph-mon[31860]: purged_snaps scrub starts 2024-03-20T08:43:52.205 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:43:51 smithi183 ceph-mon[31860]: purged_snaps scrub ok 2024-03-20T08:43:52.205 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:43:51 smithi183 ceph-mon[31860]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 85 MiB used, 268 GiB / 268 GiB avail 2024-03-20T08:43:52.205 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:43:51 smithi183 ceph-mon[31860]: osd.3 [v2:172.21.15.183:6826/4169086406,v1:172.21.15.183:6827/4169086406] boot 2024-03-20T08:43:52.205 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:43:51 smithi183 ceph-mon[31860]: osdmap e25: 4 total, 4 up, 4 in 2024-03-20T08:43:52.205 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:43:51 smithi183 ceph-mon[31860]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-20T08:43:52.264 INFO:teuthology.orchestra.run.smithi194.stderr:Inferring config /var/lib/ceph/4c3f3eba-e695-11ee-95c9-87774f69a715/mon.b/config 2024-03-20T08:43:54.138 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:43:53 smithi194 ceph-mon[27237]: osdmap e26: 4 total, 4 up, 4 in 2024-03-20T08:43:54.138 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:43:53 smithi194 ceph-mon[27237]: pgmap v105: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-03-20T08:43:54.204 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:43:53 smithi183 ceph-mon[25285]: osdmap e26: 4 total, 4 up, 4 in 2024-03-20T08:43:54.204 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:43:53 smithi183 ceph-mon[25285]: pgmap v105: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-03-20T08:43:54.204 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:43:53 smithi183 ceph-mon[31860]: osdmap e26: 4 total, 4 up, 4 in 2024-03-20T08:43:54.205 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:43:53 smithi183 ceph-mon[31860]: pgmap v105: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-03-20T08:43:55.138 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:43:54 smithi194 ceph-mon[27237]: from='client.24196 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi194:/dev/nvme4n1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-20T08:43:55.139 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:43:54 smithi194 ceph-mon[27237]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-20T08:43:55.139 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:43:54 smithi194 ceph-mon[27237]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-20T08:43:55.139 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:43:54 smithi194 ceph-mon[27237]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-20T08:43:55.204 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:43:54 smithi183 ceph-mon[25285]: from='client.24196 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi194:/dev/nvme4n1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-20T08:43:55.204 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:43:54 smithi183 ceph-mon[25285]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-20T08:43:55.204 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:43:54 smithi183 ceph-mon[25285]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-20T08:43:55.204 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:43:54 smithi183 ceph-mon[25285]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-20T08:43:55.205 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:43:54 smithi183 ceph-mon[31860]: from='client.24196 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi194:/dev/nvme4n1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-20T08:43:55.205 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:43:54 smithi183 ceph-mon[31860]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-20T08:43:55.205 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:43:54 smithi183 ceph-mon[31860]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-20T08:43:55.205 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:43:54 smithi183 ceph-mon[31860]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-20T08:43:56.139 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:43:55 smithi194 ceph-mon[27237]: pgmap v106: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-03-20T08:43:56.151 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:43:55 smithi183 ceph-mon[25285]: pgmap v106: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-03-20T08:43:56.152 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:43:55 smithi183 ceph-mon[31860]: pgmap v106: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-03-20T08:43:57.138 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:43:56 smithi194 ceph-mon[27237]: from='client.? 172.21.15.194:0/3077391646' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "38be26c8-fc1b-4f09-8545-1f58b80a38b7"}]: dispatch 2024-03-20T08:43:57.139 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:43:56 smithi194 ceph-mon[27237]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "38be26c8-fc1b-4f09-8545-1f58b80a38b7"}]: dispatch 2024-03-20T08:43:57.139 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:43:56 smithi194 ceph-mon[27237]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "38be26c8-fc1b-4f09-8545-1f58b80a38b7"}]': finished 2024-03-20T08:43:57.139 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:43:56 smithi194 ceph-mon[27237]: osdmap e27: 5 total, 4 up, 5 in 2024-03-20T08:43:57.139 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:43:56 smithi194 ceph-mon[27237]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-20T08:43:57.145 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:43:56 smithi183 ceph-mon[25285]: from='client.? 172.21.15.194:0/3077391646' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "38be26c8-fc1b-4f09-8545-1f58b80a38b7"}]: dispatch 2024-03-20T08:43:57.145 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:43:56 smithi183 ceph-mon[25285]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "38be26c8-fc1b-4f09-8545-1f58b80a38b7"}]: dispatch 2024-03-20T08:43:57.146 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:43:56 smithi183 ceph-mon[25285]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "38be26c8-fc1b-4f09-8545-1f58b80a38b7"}]': finished 2024-03-20T08:43:57.146 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:43:56 smithi183 ceph-mon[25285]: osdmap e27: 5 total, 4 up, 5 in 2024-03-20T08:43:57.146 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:43:56 smithi183 ceph-mon[25285]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-20T08:43:57.146 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:43:56 smithi183 ceph-mon[31860]: from='client.? 172.21.15.194:0/3077391646' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "38be26c8-fc1b-4f09-8545-1f58b80a38b7"}]: dispatch 2024-03-20T08:43:57.146 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:43:56 smithi183 ceph-mon[31860]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "38be26c8-fc1b-4f09-8545-1f58b80a38b7"}]: dispatch 2024-03-20T08:43:57.146 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:43:56 smithi183 ceph-mon[31860]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "38be26c8-fc1b-4f09-8545-1f58b80a38b7"}]': finished 2024-03-20T08:43:57.146 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:43:56 smithi183 ceph-mon[31860]: osdmap e27: 5 total, 4 up, 5 in 2024-03-20T08:43:57.146 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:43:56 smithi183 ceph-mon[31860]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-20T08:43:58.138 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:43:57 smithi194 ceph-mon[27237]: pgmap v108: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-03-20T08:43:58.138 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:43:57 smithi194 ceph-mon[27237]: from='client.? 172.21.15.194:0/3234645966' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-20T08:43:58.204 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:43:57 smithi183 ceph-mon[25285]: pgmap v108: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-03-20T08:43:58.204 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:43:57 smithi183 ceph-mon[25285]: from='client.? 172.21.15.194:0/3234645966' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-20T08:43:58.204 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:43:57 smithi183 ceph-mon[31860]: pgmap v108: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-03-20T08:43:58.205 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:43:57 smithi183 ceph-mon[31860]: from='client.? 172.21.15.194:0/3234645966' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-20T08:43:59.138 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:43:58 smithi194 ceph-mon[27237]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:43:59.139 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:43:58 smithi194 ceph-mon[27237]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:43:59.139 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:43:58 smithi194 ceph-mon[27237]: Detected new or changed devices on smithi183 2024-03-20T08:43:59.139 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:43:58 smithi194 ceph-mon[27237]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:43:59.204 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:43:58 smithi183 ceph-mon[25285]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:43:59.204 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:43:58 smithi183 ceph-mon[25285]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:43:59.204 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:43:58 smithi183 ceph-mon[25285]: Detected new or changed devices on smithi183 2024-03-20T08:43:59.204 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:43:58 smithi183 ceph-mon[25285]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:43:59.205 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:43:58 smithi183 ceph-mon[31860]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:43:59.205 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:43:58 smithi183 ceph-mon[31860]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:43:59.205 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:43:58 smithi183 ceph-mon[31860]: Detected new or changed devices on smithi183 2024-03-20T08:43:59.205 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:43:58 smithi183 ceph-mon[31860]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:44:00.138 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:43:59 smithi194 ceph-mon[27237]: pgmap v109: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-03-20T08:44:00.204 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:43:59 smithi183 ceph-mon[25285]: pgmap v109: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-03-20T08:44:00.204 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:43:59 smithi183 ceph-mon[31860]: pgmap v109: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-03-20T08:44:02.214 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:44:02 smithi194 ceph-mon[27237]: pgmap v110: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-03-20T08:44:02.454 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:44:02 smithi183 ceph-mon[25285]: pgmap v110: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-03-20T08:44:02.454 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:44:02 smithi183 ceph-mon[31860]: pgmap v110: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-03-20T08:44:04.388 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:44:04 smithi194 ceph-mon[27237]: pgmap v111: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-03-20T08:44:04.389 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:44:04 smithi194 ceph-mon[27237]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-03-20T08:44:04.389 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:44:04 smithi194 ceph-mon[27237]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-20T08:44:04.389 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:44:04 smithi194 ceph-mon[27237]: Deploying daemon osd.4 on smithi194 2024-03-20T08:44:04.454 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:44:04 smithi183 ceph-mon[25285]: pgmap v111: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-03-20T08:44:04.455 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:44:04 smithi183 ceph-mon[25285]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-03-20T08:44:04.455 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:44:04 smithi183 ceph-mon[25285]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-20T08:44:04.455 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:44:04 smithi183 ceph-mon[25285]: Deploying daemon osd.4 on smithi194 2024-03-20T08:44:04.455 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:44:04 smithi183 ceph-mon[31860]: pgmap v111: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-03-20T08:44:04.456 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:44:04 smithi183 ceph-mon[31860]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-03-20T08:44:04.456 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:44:04 smithi183 ceph-mon[31860]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-20T08:44:04.456 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:44:04 smithi183 ceph-mon[31860]: Deploying daemon osd.4 on smithi194 2024-03-20T08:44:06.321 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:44:06 smithi194 ceph-mon[27237]: pgmap v112: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-03-20T08:44:06.454 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:44:06 smithi183 ceph-mon[25285]: pgmap v112: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-03-20T08:44:06.454 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:44:06 smithi183 ceph-mon[31860]: pgmap v112: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-03-20T08:44:07.755 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:44:07 smithi194 ceph-mon[27237]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:44:07.756 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:44:07 smithi194 ceph-mon[27237]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:44:07.756 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:44:07 smithi194 ceph-mon[27237]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:44:07.756 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:44:07 smithi194 ceph-mon[27237]: pgmap v113: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-03-20T08:44:07.756 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:44:07 smithi194 ceph-mon[27237]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:44:07.756 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:44:07 smithi194 ceph-mon[27237]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:44:07.756 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:44:07 smithi194 ceph-mon[27237]: pgmap v114: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-03-20T08:44:07.756 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:44:07 smithi194 ceph-mon[27237]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:44:07.756 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:44:07 smithi194 ceph-mon[27237]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-20T08:44:07.756 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:44:07 smithi194 ceph-mon[27237]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-20T08:44:07.756 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:44:07 smithi194 ceph-mon[27237]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-20T08:44:07.756 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:44:07 smithi194 ceph-mon[27237]: pgmap v115: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-03-20T08:44:07.756 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:44:07 smithi194 ceph-mon[27237]: pgmap v116: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-03-20T08:44:07.757 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:44:07 smithi194 ceph-mon[27237]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:44:08.205 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:44:07 smithi183 ceph-mon[25285]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:44:08.205 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:44:07 smithi183 ceph-mon[25285]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:44:08.205 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:44:07 smithi183 ceph-mon[25285]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:44:08.205 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:44:07 smithi183 ceph-mon[25285]: pgmap v113: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-03-20T08:44:08.205 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:44:07 smithi183 ceph-mon[25285]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:44:08.205 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:44:07 smithi183 ceph-mon[25285]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:44:08.205 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:44:07 smithi183 ceph-mon[25285]: pgmap v114: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-03-20T08:44:08.205 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:44:07 smithi183 ceph-mon[25285]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:44:08.205 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:44:07 smithi183 ceph-mon[25285]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-20T08:44:08.205 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:44:07 smithi183 ceph-mon[25285]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-20T08:44:08.205 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:44:07 smithi183 ceph-mon[25285]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-20T08:44:08.205 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:44:07 smithi183 ceph-mon[25285]: pgmap v115: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-03-20T08:44:08.205 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:44:07 smithi183 ceph-mon[25285]: pgmap v116: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-03-20T08:44:08.206 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:44:07 smithi183 ceph-mon[25285]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:44:08.206 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:44:07 smithi183 ceph-mon[31860]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:44:08.206 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:44:07 smithi183 ceph-mon[31860]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:44:08.206 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:44:07 smithi183 ceph-mon[31860]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:44:08.206 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:44:07 smithi183 ceph-mon[31860]: pgmap v113: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-03-20T08:44:08.206 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:44:07 smithi183 ceph-mon[31860]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:44:08.206 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:44:07 smithi183 ceph-mon[31860]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:44:08.206 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:44:07 smithi183 ceph-mon[31860]: pgmap v114: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-03-20T08:44:08.206 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:44:07 smithi183 ceph-mon[31860]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:44:08.206 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:44:07 smithi183 ceph-mon[31860]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-20T08:44:08.206 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:44:07 smithi183 ceph-mon[31860]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-20T08:44:08.206 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:44:07 smithi183 ceph-mon[31860]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-20T08:44:08.207 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:44:07 smithi183 ceph-mon[31860]: pgmap v115: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-03-20T08:44:08.207 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:44:07 smithi183 ceph-mon[31860]: pgmap v116: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-03-20T08:44:08.207 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:44:07 smithi183 ceph-mon[31860]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:44:08.987 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:44:08 smithi194 ceph-mon[27237]: Health check failed: 1 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2024-03-20T08:44:08.987 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:44:08 smithi194 ceph-mon[27237]: Reconfiguring osd.4 (unknown last config time)... 2024-03-20T08:44:08.987 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:44:08 smithi194 ceph-mon[27237]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-03-20T08:44:08.988 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:44:08 smithi194 ceph-mon[27237]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-20T08:44:08.988 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:44:08 smithi194 ceph-mon[27237]: Reconfiguring daemon osd.4 on smithi194 2024-03-20T08:44:09.204 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:44:08 smithi183 ceph-mon[25285]: Health check failed: 1 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2024-03-20T08:44:09.204 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:44:08 smithi183 ceph-mon[25285]: Reconfiguring osd.4 (unknown last config time)... 2024-03-20T08:44:09.204 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:44:08 smithi183 ceph-mon[25285]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-03-20T08:44:09.204 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:44:08 smithi183 ceph-mon[25285]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-20T08:44:09.204 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:44:08 smithi183 ceph-mon[25285]: Reconfiguring daemon osd.4 on smithi194 2024-03-20T08:44:09.205 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:44:08 smithi183 ceph-mon[31860]: Health check failed: 1 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2024-03-20T08:44:09.205 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:44:08 smithi183 ceph-mon[31860]: Reconfiguring osd.4 (unknown last config time)... 2024-03-20T08:44:09.205 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:44:08 smithi183 ceph-mon[31860]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-03-20T08:44:09.205 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:44:08 smithi183 ceph-mon[31860]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-20T08:44:09.205 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:44:08 smithi183 ceph-mon[31860]: Reconfiguring daemon osd.4 on smithi194 2024-03-20T08:44:10.888 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:44:10 smithi194 ceph-mon[27237]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:44:10.889 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:44:10 smithi194 ceph-mon[27237]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:44:10.889 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:44:10 smithi194 ceph-mon[27237]: pgmap v117: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-03-20T08:44:10.955 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:44:10 smithi183 ceph-mon[25285]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:44:10.955 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:44:10 smithi183 ceph-mon[25285]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:44:10.955 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:44:10 smithi183 ceph-mon[25285]: pgmap v117: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-03-20T08:44:10.955 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:44:10 smithi183 ceph-mon[31860]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:44:10.955 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:44:10 smithi183 ceph-mon[31860]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:44:10.955 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:44:10 smithi183 ceph-mon[31860]: pgmap v117: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-03-20T08:44:11.882 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:44:11 smithi194 ceph-mon[27237]: pgmap v118: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-03-20T08:44:12.204 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:44:11 smithi183 ceph-mon[25285]: pgmap v118: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-03-20T08:44:12.204 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:44:11 smithi183 ceph-mon[31860]: pgmap v118: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-03-20T08:44:12.765 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:44:12 smithi194 ceph-mon[27237]: from='osd.4 [v2:172.21.15.194:6800/2756833656,v1:172.21.15.194:6801/2756833656]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-20T08:44:12.765 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:44:12 smithi194 ceph-mon[27237]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-20T08:44:13.100 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:44:12 smithi183 ceph-mon[25285]: from='osd.4 [v2:172.21.15.194:6800/2756833656,v1:172.21.15.194:6801/2756833656]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-20T08:44:13.101 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:44:12 smithi183 ceph-mon[25285]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-20T08:44:13.101 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:44:12 smithi183 ceph-mon[31860]: from='osd.4 [v2:172.21.15.194:6800/2756833656,v1:172.21.15.194:6801/2756833656]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-20T08:44:13.101 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:44:12 smithi183 ceph-mon[31860]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-20T08:44:13.124 INFO:teuthology.orchestra.run.smithi194.stdout:Created osd(s) 4 on host 'smithi194' 2024-03-20T08:44:13.794 DEBUG:teuthology.orchestra.run.smithi194:osd.4> sudo journalctl -f -n 0 -u ceph-4c3f3eba-e695-11ee-95c9-87774f69a715@osd.4.service 2024-03-20T08:44:13.796 INFO:tasks.cephadm:Deploying osd.5 on smithi194 with /dev/nvme3n1... 2024-03-20T08:44:13.796 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4c3f3eba-e695-11ee-95c9-87774f69a715 -- ceph-volume lvm zap /dev/nvme3n1 2024-03-20T08:44:14.014 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:44:13 smithi194 ceph-mon[27237]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-03-20T08:44:14.015 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:44:13 smithi194 ceph-mon[27237]: from='osd.4 [v2:172.21.15.194:6800/2756833656,v1:172.21.15.194:6801/2756833656]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi194", "root=default"]}]: dispatch 2024-03-20T08:44:14.015 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:44:13 smithi194 ceph-mon[27237]: osdmap e28: 5 total, 4 up, 5 in 2024-03-20T08:44:14.015 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:44:13 smithi194 ceph-mon[27237]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-20T08:44:14.015 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:44:13 smithi194 ceph-mon[27237]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi194", "root=default"]}]: dispatch 2024-03-20T08:44:14.015 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:44:13 smithi194 ceph-mon[27237]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:44:14.015 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:44:13 smithi194 ceph-mon[27237]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:44:14.015 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:44:13 smithi194 ceph-mon[27237]: pgmap v120: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-03-20T08:44:14.204 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:44:13 smithi183 ceph-mon[25285]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-03-20T08:44:14.204 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:44:13 smithi183 ceph-mon[25285]: from='osd.4 [v2:172.21.15.194:6800/2756833656,v1:172.21.15.194:6801/2756833656]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi194", "root=default"]}]: dispatch 2024-03-20T08:44:14.204 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:44:13 smithi183 ceph-mon[25285]: osdmap e28: 5 total, 4 up, 5 in 2024-03-20T08:44:14.204 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:44:13 smithi183 ceph-mon[25285]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-20T08:44:14.205 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:44:13 smithi183 ceph-mon[25285]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi194", "root=default"]}]: dispatch 2024-03-20T08:44:14.205 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:44:13 smithi183 ceph-mon[25285]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:44:14.205 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:44:13 smithi183 ceph-mon[25285]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:44:14.205 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:44:13 smithi183 ceph-mon[25285]: pgmap v120: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-03-20T08:44:14.205 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:44:13 smithi183 ceph-mon[31860]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-03-20T08:44:14.205 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:44:13 smithi183 ceph-mon[31860]: from='osd.4 [v2:172.21.15.194:6800/2756833656,v1:172.21.15.194:6801/2756833656]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi194", "root=default"]}]: dispatch 2024-03-20T08:44:14.205 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:44:13 smithi183 ceph-mon[31860]: osdmap e28: 5 total, 4 up, 5 in 2024-03-20T08:44:14.205 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:44:13 smithi183 ceph-mon[31860]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-20T08:44:14.205 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:44:13 smithi183 ceph-mon[31860]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi194", "root=default"]}]: dispatch 2024-03-20T08:44:14.205 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:44:13 smithi183 ceph-mon[31860]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:44:14.205 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:44:13 smithi183 ceph-mon[31860]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:44:14.205 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:44:13 smithi183 ceph-mon[31860]: pgmap v120: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-03-20T08:44:14.407 INFO:teuthology.orchestra.run.smithi194.stderr:Inferring config /var/lib/ceph/4c3f3eba-e695-11ee-95c9-87774f69a715/mon.b/config 2024-03-20T08:44:14.792 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:44:14 smithi194 ceph-mon[27237]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi194", "root=default"]}]': finished 2024-03-20T08:44:14.792 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:44:14 smithi194 ceph-mon[27237]: osdmap e29: 5 total, 4 up, 5 in 2024-03-20T08:44:14.792 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:44:14 smithi194 ceph-mon[27237]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-20T08:44:14.792 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:44:14 smithi194 ceph-mon[27237]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-20T08:44:15.204 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:44:14 smithi183 ceph-mon[25285]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi194", "root=default"]}]': finished 2024-03-20T08:44:15.204 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:44:14 smithi183 ceph-mon[25285]: osdmap e29: 5 total, 4 up, 5 in 2024-03-20T08:44:15.204 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:44:14 smithi183 ceph-mon[25285]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-20T08:44:15.205 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:44:14 smithi183 ceph-mon[25285]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-20T08:44:15.205 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:44:14 smithi183 ceph-mon[31860]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi194", "root=default"]}]': finished 2024-03-20T08:44:15.205 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:44:14 smithi183 ceph-mon[31860]: osdmap e29: 5 total, 4 up, 5 in 2024-03-20T08:44:15.205 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:44:14 smithi183 ceph-mon[31860]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-20T08:44:15.205 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:44:14 smithi183 ceph-mon[31860]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-20T08:44:16.041 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:44:15 smithi183 ceph-mon[25285]: purged_snaps scrub starts 2024-03-20T08:44:16.041 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:44:15 smithi183 ceph-mon[25285]: purged_snaps scrub ok 2024-03-20T08:44:16.041 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:44:15 smithi183 ceph-mon[25285]: osd.4 [v2:172.21.15.194:6800/2756833656,v1:172.21.15.194:6801/2756833656] boot 2024-03-20T08:44:16.041 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:44:15 smithi183 ceph-mon[25285]: osdmap e30: 5 total, 5 up, 5 in 2024-03-20T08:44:16.042 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:44:15 smithi183 ceph-mon[25285]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-20T08:44:16.042 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:44:15 smithi183 ceph-mon[25285]: osdmap e31: 5 total, 5 up, 5 in 2024-03-20T08:44:16.042 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:44:15 smithi183 ceph-mon[25285]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-03-20T08:44:16.042 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:44:15 smithi183 ceph-mon[31860]: purged_snaps scrub starts 2024-03-20T08:44:16.042 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:44:15 smithi183 ceph-mon[31860]: purged_snaps scrub ok 2024-03-20T08:44:16.042 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:44:15 smithi183 ceph-mon[31860]: osd.4 [v2:172.21.15.194:6800/2756833656,v1:172.21.15.194:6801/2756833656] boot 2024-03-20T08:44:16.042 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:44:15 smithi183 ceph-mon[31860]: osdmap e30: 5 total, 5 up, 5 in 2024-03-20T08:44:16.042 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:44:15 smithi183 ceph-mon[31860]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-20T08:44:16.042 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:44:15 smithi183 ceph-mon[31860]: osdmap e31: 5 total, 5 up, 5 in 2024-03-20T08:44:16.042 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:44:15 smithi183 ceph-mon[31860]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-03-20T08:44:16.083 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:44:15 smithi194 ceph-mon[27237]: purged_snaps scrub starts 2024-03-20T08:44:16.083 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:44:15 smithi194 ceph-mon[27237]: purged_snaps scrub ok 2024-03-20T08:44:16.083 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:44:15 smithi194 ceph-mon[27237]: osd.4 [v2:172.21.15.194:6800/2756833656,v1:172.21.15.194:6801/2756833656] boot 2024-03-20T08:44:16.083 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:44:15 smithi194 ceph-mon[27237]: osdmap e30: 5 total, 5 up, 5 in 2024-03-20T08:44:16.084 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:44:15 smithi194 ceph-mon[27237]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-20T08:44:16.084 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:44:15 smithi194 ceph-mon[27237]: osdmap e31: 5 total, 5 up, 5 in 2024-03-20T08:44:16.084 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:44:15 smithi194 ceph-mon[27237]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-03-20T08:44:16.681 INFO:teuthology.orchestra.run.smithi194.stderr:--> Zapping: /dev/nvme3n1 2024-03-20T08:44:16.682 INFO:teuthology.orchestra.run.smithi194.stderr:--> --destroy was not specified, but zapping a whole device will remove the partition table 2024-03-20T08:44:16.682 INFO:teuthology.orchestra.run.smithi194.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/nvme3n1 bs=1M count=10 conv=fsync 2024-03-20T08:44:16.682 INFO:teuthology.orchestra.run.smithi194.stderr: stderr: 10+0 records in 2024-03-20T08:44:16.682 INFO:teuthology.orchestra.run.smithi194.stderr:10+0 records out 2024-03-20T08:44:16.682 INFO:teuthology.orchestra.run.smithi194.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0136044 s, 771 MB/s 2024-03-20T08:44:16.682 INFO:teuthology.orchestra.run.smithi194.stderr:--> Zapping successful for: 2024-03-20T08:44:17.247 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4c3f3eba-e695-11ee-95c9-87774f69a715 -- ceph orch daemon add osd smithi194:/dev/nvme3n1 2024-03-20T08:44:17.272 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:44:17 smithi194 ceph-mon[27237]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:44:17.272 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:44:17 smithi194 ceph-mon[27237]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:44:17.272 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:44:17 smithi194 ceph-mon[27237]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-03-20T08:44:17.272 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:44:17 smithi194 ceph-mon[27237]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:44:17.272 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:44:17 smithi194 ceph-mon[27237]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:44:17.272 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:44:17 smithi194 ceph-mon[27237]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:44:17.272 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:44:17 smithi194 ceph-mon[27237]: Health check cleared: CEPHADM_FAILED_DAEMON (was: 1 failed cephadm daemon(s)) 2024-03-20T08:44:17.272 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:44:17 smithi194 ceph-mon[27237]: Cluster is now healthy 2024-03-20T08:44:17.272 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:44:17 smithi194 ceph-mon[27237]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-20T08:44:17.272 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:44:17 smithi194 ceph-mon[27237]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-20T08:44:17.273 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:44:17 smithi194 ceph-mon[27237]: Adjusting osd_memory_target on smithi194 to 16064M 2024-03-20T08:44:17.273 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:44:17 smithi194 ceph-mon[27237]: osdmap e32: 5 total, 5 up, 5 in 2024-03-20T08:44:17.273 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:44:17 smithi194 ceph-mon[27237]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:44:17.273 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:44:17 smithi194 ceph-mon[27237]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-20T08:44:17.273 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:44:17 smithi194 ceph-mon[27237]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-20T08:44:17.273 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:44:17 smithi194 ceph-mon[27237]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:44:17.284 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:44:17 smithi183 ceph-mon[31860]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:44:17.284 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:44:17 smithi183 ceph-mon[31860]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:44:17.284 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:44:17 smithi183 ceph-mon[31860]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-03-20T08:44:17.284 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:44:17 smithi183 ceph-mon[31860]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:44:17.284 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:44:17 smithi183 ceph-mon[31860]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:44:17.284 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:44:17 smithi183 ceph-mon[31860]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:44:17.284 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:44:17 smithi183 ceph-mon[31860]: Health check cleared: CEPHADM_FAILED_DAEMON (was: 1 failed cephadm daemon(s)) 2024-03-20T08:44:17.284 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:44:17 smithi183 ceph-mon[31860]: Cluster is now healthy 2024-03-20T08:44:17.284 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:44:17 smithi183 ceph-mon[31860]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-20T08:44:17.284 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:44:17 smithi183 ceph-mon[31860]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-20T08:44:17.284 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:44:17 smithi183 ceph-mon[31860]: Adjusting osd_memory_target on smithi194 to 16064M 2024-03-20T08:44:17.285 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:44:17 smithi183 ceph-mon[31860]: osdmap e32: 5 total, 5 up, 5 in 2024-03-20T08:44:17.285 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:44:17 smithi183 ceph-mon[31860]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:44:17.285 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:44:17 smithi183 ceph-mon[31860]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-20T08:44:17.285 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:44:17 smithi183 ceph-mon[31860]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-20T08:44:17.285 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:44:17 smithi183 ceph-mon[31860]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:44:17.285 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:44:17 smithi183 ceph-mon[25285]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:44:17.285 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:44:17 smithi183 ceph-mon[25285]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:44:17.285 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:44:17 smithi183 ceph-mon[25285]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-03-20T08:44:17.285 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:44:17 smithi183 ceph-mon[25285]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:44:17.285 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:44:17 smithi183 ceph-mon[25285]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:44:17.285 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:44:17 smithi183 ceph-mon[25285]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:44:17.285 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:44:17 smithi183 ceph-mon[25285]: Health check cleared: CEPHADM_FAILED_DAEMON (was: 1 failed cephadm daemon(s)) 2024-03-20T08:44:17.286 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:44:17 smithi183 ceph-mon[25285]: Cluster is now healthy 2024-03-20T08:44:17.286 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:44:17 smithi183 ceph-mon[25285]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-20T08:44:17.286 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:44:17 smithi183 ceph-mon[25285]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-20T08:44:17.286 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:44:17 smithi183 ceph-mon[25285]: Adjusting osd_memory_target on smithi194 to 16064M 2024-03-20T08:44:17.286 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:44:17 smithi183 ceph-mon[25285]: osdmap e32: 5 total, 5 up, 5 in 2024-03-20T08:44:17.286 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:44:17 smithi183 ceph-mon[25285]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:44:17.286 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:44:17 smithi183 ceph-mon[25285]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-20T08:44:17.286 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:44:17 smithi183 ceph-mon[25285]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-20T08:44:17.286 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:44:17 smithi183 ceph-mon[25285]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:44:17.482 INFO:teuthology.orchestra.run.smithi194.stderr:Inferring config /var/lib/ceph/4c3f3eba-e695-11ee-95c9-87774f69a715/mon.b/config 2024-03-20T08:44:18.377 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:44:18 smithi194 ceph-mon[27237]: osdmap e33: 5 total, 5 up, 5 in 2024-03-20T08:44:18.704 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:44:18 smithi183 ceph-mon[25285]: osdmap e33: 5 total, 5 up, 5 in 2024-03-20T08:44:18.705 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:44:18 smithi183 ceph-mon[31860]: osdmap e33: 5 total, 5 up, 5 in 2024-03-20T08:44:19.638 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:44:19 smithi194 ceph-mon[27237]: pgmap v128: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail; 177 KiB/s, 0 objects/s recovering 2024-03-20T08:44:19.639 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:44:19 smithi194 ceph-mon[27237]: osdmap e34: 5 total, 5 up, 5 in 2024-03-20T08:44:19.639 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:44:19 smithi194 ceph-mon[27237]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-20T08:44:19.639 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:44:19 smithi194 ceph-mon[27237]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-20T08:44:19.639 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:44:19 smithi194 ceph-mon[27237]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-20T08:44:19.704 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:44:19 smithi183 ceph-mon[31860]: pgmap v128: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail; 177 KiB/s, 0 objects/s recovering 2024-03-20T08:44:19.704 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:44:19 smithi183 ceph-mon[31860]: osdmap e34: 5 total, 5 up, 5 in 2024-03-20T08:44:19.704 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:44:19 smithi183 ceph-mon[31860]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-20T08:44:19.705 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:44:19 smithi183 ceph-mon[31860]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-20T08:44:19.705 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:44:19 smithi183 ceph-mon[31860]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-20T08:44:19.705 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:44:19 smithi183 ceph-mon[25285]: pgmap v128: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail; 177 KiB/s, 0 objects/s recovering 2024-03-20T08:44:19.705 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:44:19 smithi183 ceph-mon[25285]: osdmap e34: 5 total, 5 up, 5 in 2024-03-20T08:44:19.705 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:44:19 smithi183 ceph-mon[25285]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-20T08:44:19.705 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:44:19 smithi183 ceph-mon[25285]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-20T08:44:19.705 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:44:19 smithi183 ceph-mon[25285]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-20T08:44:20.325 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:44:20 smithi194 ceph-mon[27237]: from='client.24223 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi194:/dev/nvme3n1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-20T08:44:20.704 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:44:20 smithi183 ceph-mon[25285]: from='client.24223 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi194:/dev/nvme3n1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-20T08:44:20.705 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:44:20 smithi183 ceph-mon[31860]: from='client.24223 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi194:/dev/nvme3n1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-20T08:44:21.638 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:44:21 smithi194 ceph-mon[27237]: pgmap v130: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 140 MiB used, 447 GiB / 447 GiB avail; 132 KiB/s, 0 objects/s recovering 2024-03-20T08:44:21.704 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:44:21 smithi183 ceph-mon[31860]: pgmap v130: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 140 MiB used, 447 GiB / 447 GiB avail; 132 KiB/s, 0 objects/s recovering 2024-03-20T08:44:21.705 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:44:21 smithi183 ceph-mon[25285]: pgmap v130: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 140 MiB used, 447 GiB / 447 GiB avail; 132 KiB/s, 0 objects/s recovering 2024-03-20T08:44:22.638 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:44:22 smithi194 ceph-mon[27237]: from='client.? 172.21.15.194:0/2114540730' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "dcafefba-cc63-4c52-93a8-6296eed08381"}]: dispatch 2024-03-20T08:44:22.638 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:44:22 smithi194 ceph-mon[27237]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "dcafefba-cc63-4c52-93a8-6296eed08381"}]: dispatch 2024-03-20T08:44:22.639 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:44:22 smithi194 ceph-mon[27237]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "dcafefba-cc63-4c52-93a8-6296eed08381"}]': finished 2024-03-20T08:44:22.639 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:44:22 smithi194 ceph-mon[27237]: osdmap e35: 6 total, 5 up, 6 in 2024-03-20T08:44:22.639 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:44:22 smithi194 ceph-mon[27237]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-20T08:44:22.704 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:44:22 smithi183 ceph-mon[25285]: from='client.? 172.21.15.194:0/2114540730' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "dcafefba-cc63-4c52-93a8-6296eed08381"}]: dispatch 2024-03-20T08:44:22.704 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:44:22 smithi183 ceph-mon[25285]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "dcafefba-cc63-4c52-93a8-6296eed08381"}]: dispatch 2024-03-20T08:44:22.704 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:44:22 smithi183 ceph-mon[25285]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "dcafefba-cc63-4c52-93a8-6296eed08381"}]': finished 2024-03-20T08:44:22.704 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:44:22 smithi183 ceph-mon[25285]: osdmap e35: 6 total, 5 up, 6 in 2024-03-20T08:44:22.705 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:44:22 smithi183 ceph-mon[25285]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-20T08:44:22.705 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:44:22 smithi183 ceph-mon[31860]: from='client.? 172.21.15.194:0/2114540730' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "dcafefba-cc63-4c52-93a8-6296eed08381"}]: dispatch 2024-03-20T08:44:22.705 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:44:22 smithi183 ceph-mon[31860]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "dcafefba-cc63-4c52-93a8-6296eed08381"}]: dispatch 2024-03-20T08:44:22.705 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:44:22 smithi183 ceph-mon[31860]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "dcafefba-cc63-4c52-93a8-6296eed08381"}]': finished 2024-03-20T08:44:22.705 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:44:22 smithi183 ceph-mon[31860]: osdmap e35: 6 total, 5 up, 6 in 2024-03-20T08:44:22.705 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:44:22 smithi183 ceph-mon[31860]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-20T08:44:23.510 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:44:23 smithi183 ceph-mon[25285]: pgmap v132: 1 pgs: 1 active+clean; 577 KiB data, 140 MiB used, 447 GiB / 447 GiB avail; 99 KiB/s, 0 objects/s recovering 2024-03-20T08:44:23.510 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:44:23 smithi183 ceph-mon[25285]: from='client.? 172.21.15.194:0/2895669389' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-20T08:44:23.510 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:44:23 smithi183 ceph-mon[31860]: pgmap v132: 1 pgs: 1 active+clean; 577 KiB data, 140 MiB used, 447 GiB / 447 GiB avail; 99 KiB/s, 0 objects/s recovering 2024-03-20T08:44:23.510 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:44:23 smithi183 ceph-mon[31860]: from='client.? 172.21.15.194:0/2895669389' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-20T08:44:23.638 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:44:23 smithi194 ceph-mon[27237]: pgmap v132: 1 pgs: 1 active+clean; 577 KiB data, 140 MiB used, 447 GiB / 447 GiB avail; 99 KiB/s, 0 objects/s recovering 2024-03-20T08:44:23.638 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:44:23 smithi194 ceph-mon[27237]: from='client.? 172.21.15.194:0/2895669389' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-20T08:44:25.563 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:44:25 smithi194 ceph-mon[27237]: pgmap v133: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-03-20T08:44:25.704 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:44:25 smithi183 ceph-mon[25285]: pgmap v133: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-03-20T08:44:25.704 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:44:25 smithi183 ceph-mon[31860]: pgmap v133: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-03-20T08:44:27.344 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:44:27 smithi194 ceph-mon[27237]: pgmap v134: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-03-20T08:44:27.704 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:44:27 smithi183 ceph-mon[25285]: pgmap v134: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-03-20T08:44:27.704 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:44:27 smithi183 ceph-mon[31860]: pgmap v134: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-03-20T08:44:29.078 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:44:28 smithi194 ceph-mon[27237]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:44:29.079 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:44:28 smithi194 ceph-mon[27237]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:44:29.079 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:44:28 smithi194 ceph-mon[27237]: pgmap v135: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-03-20T08:44:29.079 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:44:28 smithi194 ceph-mon[27237]: Detected new or changed devices on smithi194 2024-03-20T08:44:29.079 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:44:28 smithi194 ceph-mon[27237]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:44:29.079 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:44:28 smithi194 ceph-mon[27237]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-20T08:44:29.079 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:44:28 smithi194 ceph-mon[27237]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-20T08:44:29.079 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:44:28 smithi194 ceph-mon[27237]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-20T08:44:29.079 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:44:28 smithi194 ceph-mon[27237]: pgmap v136: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-03-20T08:44:29.079 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:44:28 smithi194 ceph-mon[27237]: pgmap v137: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-03-20T08:44:29.079 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:44:28 smithi194 ceph-mon[27237]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:44:29.079 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:44:28 smithi194 ceph-mon[27237]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-03-20T08:44:29.079 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:44:28 smithi194 ceph-mon[27237]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-20T08:44:29.079 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:44:28 smithi194 ceph-mon[27237]: Deploying daemon osd.5 on smithi194 2024-03-20T08:44:29.204 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:44:28 smithi183 ceph-mon[25285]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:44:29.204 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:44:28 smithi183 ceph-mon[25285]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:44:29.205 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:44:28 smithi183 ceph-mon[25285]: pgmap v135: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-03-20T08:44:29.205 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:44:28 smithi183 ceph-mon[25285]: Detected new or changed devices on smithi194 2024-03-20T08:44:29.205 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:44:28 smithi183 ceph-mon[25285]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:44:29.205 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:44:28 smithi183 ceph-mon[25285]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-20T08:44:29.205 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:44:28 smithi183 ceph-mon[25285]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-20T08:44:29.205 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:44:28 smithi183 ceph-mon[25285]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-20T08:44:29.205 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:44:28 smithi183 ceph-mon[25285]: pgmap v136: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-03-20T08:44:29.205 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:44:28 smithi183 ceph-mon[25285]: pgmap v137: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-03-20T08:44:29.205 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:44:28 smithi183 ceph-mon[25285]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:44:29.205 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:44:28 smithi183 ceph-mon[25285]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-03-20T08:44:29.205 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:44:28 smithi183 ceph-mon[25285]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-20T08:44:29.205 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:44:28 smithi183 ceph-mon[25285]: Deploying daemon osd.5 on smithi194 2024-03-20T08:44:29.206 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:44:28 smithi183 ceph-mon[31860]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:44:29.206 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:44:28 smithi183 ceph-mon[31860]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:44:29.206 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:44:28 smithi183 ceph-mon[31860]: pgmap v135: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-03-20T08:44:29.206 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:44:28 smithi183 ceph-mon[31860]: Detected new or changed devices on smithi194 2024-03-20T08:44:29.206 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:44:28 smithi183 ceph-mon[31860]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:44:29.206 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:44:28 smithi183 ceph-mon[31860]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-20T08:44:29.206 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:44:28 smithi183 ceph-mon[31860]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-20T08:44:29.206 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:44:28 smithi183 ceph-mon[31860]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-20T08:44:29.206 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:44:28 smithi183 ceph-mon[31860]: pgmap v136: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-03-20T08:44:29.206 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:44:28 smithi183 ceph-mon[31860]: pgmap v137: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-03-20T08:44:29.206 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:44:28 smithi183 ceph-mon[31860]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:44:29.206 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:44:28 smithi183 ceph-mon[31860]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-03-20T08:44:29.206 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:44:28 smithi183 ceph-mon[31860]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-20T08:44:29.206 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:44:28 smithi183 ceph-mon[31860]: Deploying daemon osd.5 on smithi194 2024-03-20T08:44:29.973 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:44:29 smithi194 ceph-mon[27237]: Health check failed: 1 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2024-03-20T08:44:30.204 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:44:29 smithi183 ceph-mon[25285]: Health check failed: 1 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2024-03-20T08:44:30.204 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:44:29 smithi183 ceph-mon[31860]: Health check failed: 1 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2024-03-20T08:44:31.565 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:44:31 smithi194 ceph-mon[27237]: pgmap v138: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-03-20T08:44:31.565 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:44:31 smithi194 ceph-mon[27237]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:44:31.565 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:44:31 smithi194 ceph-mon[27237]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:44:31.565 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:44:31 smithi194 ceph-mon[27237]: pgmap v139: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-03-20T08:44:31.565 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:44:31 smithi194 ceph-mon[27237]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:44:31.565 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:44:31 smithi194 ceph-mon[27237]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-20T08:44:31.565 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:44:31 smithi194 ceph-mon[27237]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-20T08:44:31.565 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:44:31 smithi194 ceph-mon[27237]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-20T08:44:31.565 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:44:31 smithi194 ceph-mon[27237]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:44:31.565 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:44:31 smithi194 ceph-mon[27237]: Health check cleared: CEPHADM_FAILED_DAEMON (was: 1 failed cephadm daemon(s)) 2024-03-20T08:44:31.565 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:44:31 smithi194 ceph-mon[27237]: Cluster is now healthy 2024-03-20T08:44:31.704 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:44:31 smithi183 ceph-mon[25285]: pgmap v138: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-03-20T08:44:31.705 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:44:31 smithi183 ceph-mon[25285]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:44:31.705 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:44:31 smithi183 ceph-mon[25285]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:44:31.705 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:44:31 smithi183 ceph-mon[25285]: pgmap v139: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-03-20T08:44:31.705 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:44:31 smithi183 ceph-mon[25285]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:44:31.705 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:44:31 smithi183 ceph-mon[25285]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-20T08:44:31.705 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:44:31 smithi183 ceph-mon[25285]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-20T08:44:31.705 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:44:31 smithi183 ceph-mon[25285]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-20T08:44:31.705 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:44:31 smithi183 ceph-mon[25285]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:44:31.705 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:44:31 smithi183 ceph-mon[25285]: Health check cleared: CEPHADM_FAILED_DAEMON (was: 1 failed cephadm daemon(s)) 2024-03-20T08:44:31.705 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:44:31 smithi183 ceph-mon[25285]: Cluster is now healthy 2024-03-20T08:44:31.705 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:44:31 smithi183 ceph-mon[31860]: pgmap v138: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-03-20T08:44:31.706 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:44:31 smithi183 ceph-mon[31860]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:44:31.706 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:44:31 smithi183 ceph-mon[31860]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:44:31.706 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:44:31 smithi183 ceph-mon[31860]: pgmap v139: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-03-20T08:44:31.706 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:44:31 smithi183 ceph-mon[31860]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:44:31.706 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:44:31 smithi183 ceph-mon[31860]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-20T08:44:31.706 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:44:31 smithi183 ceph-mon[31860]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-20T08:44:31.706 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:44:31 smithi183 ceph-mon[31860]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-20T08:44:31.706 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:44:31 smithi183 ceph-mon[31860]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:44:31.706 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:44:31 smithi183 ceph-mon[31860]: Health check cleared: CEPHADM_FAILED_DAEMON (was: 1 failed cephadm daemon(s)) 2024-03-20T08:44:31.706 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:44:31 smithi183 ceph-mon[31860]: Cluster is now healthy 2024-03-20T08:44:33.516 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:44:33 smithi194 ceph-mon[27237]: pgmap v140: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-03-20T08:44:33.704 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:44:33 smithi183 ceph-mon[25285]: pgmap v140: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-03-20T08:44:33.704 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:44:33 smithi183 ceph-mon[31860]: pgmap v140: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-03-20T08:44:35.510 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:44:35 smithi194 ceph-mon[27237]: pgmap v141: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-03-20T08:44:35.510 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:44:35 smithi194 ceph-mon[27237]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:44:35.510 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:44:35 smithi194 ceph-mon[27237]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:44:35.510 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:44:35 smithi194 ceph-mon[27237]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-20T08:44:35.510 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:44:35 smithi194 ceph-mon[27237]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-20T08:44:35.510 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:44:35 smithi194 ceph-mon[27237]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-20T08:44:35.510 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:44:35 smithi194 ceph-mon[27237]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:44:35.704 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:44:35 smithi183 ceph-mon[25285]: pgmap v141: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-03-20T08:44:35.704 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:44:35 smithi183 ceph-mon[25285]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:44:35.705 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:44:35 smithi183 ceph-mon[25285]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:44:35.705 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:44:35 smithi183 ceph-mon[25285]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-20T08:44:35.705 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:44:35 smithi183 ceph-mon[25285]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-20T08:44:35.705 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:44:35 smithi183 ceph-mon[25285]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-20T08:44:35.705 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:44:35 smithi183 ceph-mon[25285]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:44:35.705 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:44:35 smithi183 ceph-mon[31860]: pgmap v141: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-03-20T08:44:35.705 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:44:35 smithi183 ceph-mon[31860]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:44:35.705 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:44:35 smithi183 ceph-mon[31860]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:44:35.705 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:44:35 smithi183 ceph-mon[31860]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-20T08:44:35.705 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:44:35 smithi183 ceph-mon[31860]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-20T08:44:35.705 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:44:35 smithi183 ceph-mon[31860]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-20T08:44:35.705 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:44:35 smithi183 ceph-mon[31860]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:44:37.260 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:44:37 smithi194 ceph-mon[27237]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:44:37.260 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:44:37 smithi194 ceph-mon[27237]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:44:37.260 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:44:37 smithi194 ceph-mon[27237]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:44:37.260 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:44:37 smithi194 ceph-mon[27237]: pgmap v142: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-03-20T08:44:37.454 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:44:37 smithi183 ceph-mon[25285]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:44:37.455 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:44:37 smithi183 ceph-mon[25285]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:44:37.455 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:44:37 smithi183 ceph-mon[25285]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:44:37.455 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:44:37 smithi183 ceph-mon[25285]: pgmap v142: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-03-20T08:44:37.455 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:44:37 smithi183 ceph-mon[31860]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:44:37.455 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:44:37 smithi183 ceph-mon[31860]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:44:37.455 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:44:37 smithi183 ceph-mon[31860]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:44:37.455 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:44:37 smithi183 ceph-mon[31860]: pgmap v142: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-03-20T08:44:38.548 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:44:38 smithi194 ceph-mon[27237]: from='osd.5 [v2:172.21.15.194:6808/1048900641,v1:172.21.15.194:6809/1048900641]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-20T08:44:38.549 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:44:38 smithi194 ceph-mon[27237]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-20T08:44:38.581 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:44:38 smithi183 ceph-mon[25285]: from='osd.5 [v2:172.21.15.194:6808/1048900641,v1:172.21.15.194:6809/1048900641]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-20T08:44:38.581 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:44:38 smithi183 ceph-mon[25285]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-20T08:44:38.581 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:44:38 smithi183 ceph-mon[31860]: from='osd.5 [v2:172.21.15.194:6808/1048900641,v1:172.21.15.194:6809/1048900641]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-20T08:44:38.581 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:44:38 smithi183 ceph-mon[31860]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-20T08:44:38.856 INFO:teuthology.orchestra.run.smithi194.stdout:Created osd(s) 5 on host 'smithi194' 2024-03-20T08:44:39.428 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:44:39 smithi194 ceph-mon[27237]: pgmap v143: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-03-20T08:44:39.428 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:44:39 smithi194 ceph-mon[27237]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-03-20T08:44:39.428 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:44:39 smithi194 ceph-mon[27237]: osdmap e36: 6 total, 5 up, 6 in 2024-03-20T08:44:39.428 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:44:39 smithi194 ceph-mon[27237]: from='osd.5 [v2:172.21.15.194:6808/1048900641,v1:172.21.15.194:6809/1048900641]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi194", "root=default"]}]: dispatch 2024-03-20T08:44:39.428 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:44:39 smithi194 ceph-mon[27237]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-20T08:44:39.428 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:44:39 smithi194 ceph-mon[27237]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi194", "root=default"]}]: dispatch 2024-03-20T08:44:39.428 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:44:39 smithi194 ceph-mon[27237]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:44:39.428 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:44:39 smithi194 ceph-mon[27237]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:44:39.428 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:44:39 smithi194 ceph-mon[27237]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-20T08:44:39.429 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:44:39 smithi194 ceph-mon[27237]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-20T08:44:39.429 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:44:39 smithi194 ceph-mon[27237]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-20T08:44:39.429 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:44:39 smithi194 ceph-mon[27237]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:44:39.429 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:44:39 smithi194 ceph-mon[27237]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-20T08:44:39.429 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:44:39 smithi194 ceph-mon[27237]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-20T08:44:39.429 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:44:39 smithi194 ceph-mon[27237]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:44:39.429 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:44:39 smithi194 ceph-mon[27237]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi194", "root=default"]}]': finished 2024-03-20T08:44:39.429 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:44:39 smithi194 ceph-mon[27237]: osdmap e37: 6 total, 5 up, 6 in 2024-03-20T08:44:39.459 DEBUG:teuthology.orchestra.run.smithi194:osd.5> sudo journalctl -f -n 0 -u ceph-4c3f3eba-e695-11ee-95c9-87774f69a715@osd.5.service 2024-03-20T08:44:39.502 INFO:tasks.cephadm:Deploying osd.6 on smithi194 with /dev/nvme2n1... 2024-03-20T08:44:39.502 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4c3f3eba-e695-11ee-95c9-87774f69a715 -- ceph-volume lvm zap /dev/nvme2n1 2024-03-20T08:44:39.704 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:44:39 smithi183 ceph-mon[31860]: pgmap v143: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-03-20T08:44:39.704 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:44:39 smithi183 ceph-mon[31860]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-03-20T08:44:39.704 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:44:39 smithi183 ceph-mon[31860]: osdmap e36: 6 total, 5 up, 6 in 2024-03-20T08:44:39.705 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:44:39 smithi183 ceph-mon[31860]: from='osd.5 [v2:172.21.15.194:6808/1048900641,v1:172.21.15.194:6809/1048900641]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi194", "root=default"]}]: dispatch 2024-03-20T08:44:39.705 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:44:39 smithi183 ceph-mon[31860]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-20T08:44:39.705 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:44:39 smithi183 ceph-mon[31860]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi194", "root=default"]}]: dispatch 2024-03-20T08:44:39.705 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:44:39 smithi183 ceph-mon[31860]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:44:39.705 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:44:39 smithi183 ceph-mon[31860]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:44:39.705 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:44:39 smithi183 ceph-mon[31860]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-20T08:44:39.705 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:44:39 smithi183 ceph-mon[31860]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-20T08:44:39.705 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:44:39 smithi183 ceph-mon[31860]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-20T08:44:39.705 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:44:39 smithi183 ceph-mon[31860]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:44:39.705 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:44:39 smithi183 ceph-mon[31860]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-20T08:44:39.705 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:44:39 smithi183 ceph-mon[31860]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-20T08:44:39.705 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:44:39 smithi183 ceph-mon[31860]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:44:39.705 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:44:39 smithi183 ceph-mon[31860]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi194", "root=default"]}]': finished 2024-03-20T08:44:39.705 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:44:39 smithi183 ceph-mon[31860]: osdmap e37: 6 total, 5 up, 6 in 2024-03-20T08:44:39.706 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:44:39 smithi183 ceph-mon[25285]: pgmap v143: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-03-20T08:44:39.706 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:44:39 smithi183 ceph-mon[25285]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-03-20T08:44:39.706 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:44:39 smithi183 ceph-mon[25285]: osdmap e36: 6 total, 5 up, 6 in 2024-03-20T08:44:39.706 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:44:39 smithi183 ceph-mon[25285]: from='osd.5 [v2:172.21.15.194:6808/1048900641,v1:172.21.15.194:6809/1048900641]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi194", "root=default"]}]: dispatch 2024-03-20T08:44:39.706 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:44:39 smithi183 ceph-mon[25285]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-20T08:44:39.706 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:44:39 smithi183 ceph-mon[25285]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi194", "root=default"]}]: dispatch 2024-03-20T08:44:39.706 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:44:39 smithi183 ceph-mon[25285]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:44:39.706 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:44:39 smithi183 ceph-mon[25285]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:44:39.706 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:44:39 smithi183 ceph-mon[25285]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-20T08:44:39.706 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:44:39 smithi183 ceph-mon[25285]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-20T08:44:39.706 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:44:39 smithi183 ceph-mon[25285]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-20T08:44:39.706 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:44:39 smithi183 ceph-mon[25285]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:44:39.706 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:44:39 smithi183 ceph-mon[25285]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-20T08:44:39.707 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:44:39 smithi183 ceph-mon[25285]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-20T08:44:39.707 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:44:39 smithi183 ceph-mon[25285]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:44:39.707 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:44:39 smithi183 ceph-mon[25285]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi194", "root=default"]}]': finished 2024-03-20T08:44:39.707 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:44:39 smithi183 ceph-mon[25285]: osdmap e37: 6 total, 5 up, 6 in 2024-03-20T08:44:39.796 INFO:teuthology.orchestra.run.smithi194.stderr:Inferring config /var/lib/ceph/4c3f3eba-e695-11ee-95c9-87774f69a715/mon.b/config 2024-03-20T08:44:39.993 INFO:journalctl@ceph.osd.5.smithi194.stdout:Mar 20 08:44:39 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-osd-5[37760]: 2024-03-20T08:44:39.741+0000 7f6098abc700 -1 osd.5 0 waiting for initial osdmap 2024-03-20T08:44:39.993 INFO:journalctl@ceph.osd.5.smithi194.stdout:Mar 20 08:44:39 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-osd-5[37760]: 2024-03-20T08:44:39.749+0000 7f60908a7700 -1 osd.5 37 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-20T08:44:40.638 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:44:40 smithi194 ceph-mon[27237]: Adjusting osd_memory_target on smithi194 to 8032M 2024-03-20T08:44:40.639 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:44:40 smithi194 ceph-mon[27237]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-20T08:44:40.639 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:44:40 smithi194 ceph-mon[27237]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-20T08:44:40.639 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:44:40 smithi194 ceph-mon[27237]: from='osd.5 ' entity='osd.5' 2024-03-20T08:44:40.639 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:44:40 smithi194 ceph-mon[27237]: osd.5 [v2:172.21.15.194:6808/1048900641,v1:172.21.15.194:6809/1048900641] boot 2024-03-20T08:44:40.639 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:44:40 smithi194 ceph-mon[27237]: osdmap e38: 6 total, 6 up, 6 in 2024-03-20T08:44:40.639 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:44:40 smithi194 ceph-mon[27237]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-20T08:44:40.704 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:44:40 smithi183 ceph-mon[25285]: Adjusting osd_memory_target on smithi194 to 8032M 2024-03-20T08:44:40.704 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:44:40 smithi183 ceph-mon[25285]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-20T08:44:40.704 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:44:40 smithi183 ceph-mon[25285]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-20T08:44:40.705 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:44:40 smithi183 ceph-mon[25285]: from='osd.5 ' entity='osd.5' 2024-03-20T08:44:40.705 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:44:40 smithi183 ceph-mon[25285]: osd.5 [v2:172.21.15.194:6808/1048900641,v1:172.21.15.194:6809/1048900641] boot 2024-03-20T08:44:40.705 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:44:40 smithi183 ceph-mon[25285]: osdmap e38: 6 total, 6 up, 6 in 2024-03-20T08:44:40.705 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:44:40 smithi183 ceph-mon[25285]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-20T08:44:40.705 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:44:40 smithi183 ceph-mon[31860]: Adjusting osd_memory_target on smithi194 to 8032M 2024-03-20T08:44:40.705 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:44:40 smithi183 ceph-mon[31860]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-20T08:44:40.705 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:44:40 smithi183 ceph-mon[31860]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-20T08:44:40.705 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:44:40 smithi183 ceph-mon[31860]: from='osd.5 ' entity='osd.5' 2024-03-20T08:44:40.705 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:44:40 smithi183 ceph-mon[31860]: osd.5 [v2:172.21.15.194:6808/1048900641,v1:172.21.15.194:6809/1048900641] boot 2024-03-20T08:44:40.705 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:44:40 smithi183 ceph-mon[31860]: osdmap e38: 6 total, 6 up, 6 in 2024-03-20T08:44:40.705 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:44:40 smithi183 ceph-mon[31860]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-20T08:44:41.603 INFO:teuthology.orchestra.run.smithi194.stderr:--> Zapping: /dev/nvme2n1 2024-03-20T08:44:41.603 INFO:teuthology.orchestra.run.smithi194.stderr:--> --destroy was not specified, but zapping a whole device will remove the partition table 2024-03-20T08:44:41.603 INFO:teuthology.orchestra.run.smithi194.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/nvme2n1 bs=1M count=10 conv=fsync 2024-03-20T08:44:41.603 INFO:teuthology.orchestra.run.smithi194.stderr: stderr: 10+0 records in 2024-03-20T08:44:41.603 INFO:teuthology.orchestra.run.smithi194.stderr:10+0 records out 2024-03-20T08:44:41.604 INFO:teuthology.orchestra.run.smithi194.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0145467 s, 721 MB/s 2024-03-20T08:44:41.604 INFO:teuthology.orchestra.run.smithi194.stderr:--> Zapping successful for: 2024-03-20T08:44:41.604 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:44:41 smithi194 ceph-mon[27237]: purged_snaps scrub starts 2024-03-20T08:44:41.604 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:44:41 smithi194 ceph-mon[27237]: purged_snaps scrub ok 2024-03-20T08:44:41.604 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:44:41 smithi194 ceph-mon[27237]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-03-20T08:44:41.604 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:44:41 smithi194 ceph-mon[27237]: osdmap e39: 6 total, 6 up, 6 in 2024-03-20T08:44:41.605 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:44:41 smithi183 ceph-mon[25285]: purged_snaps scrub starts 2024-03-20T08:44:41.605 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:44:41 smithi183 ceph-mon[25285]: purged_snaps scrub ok 2024-03-20T08:44:41.606 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:44:41 smithi183 ceph-mon[25285]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-03-20T08:44:41.606 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:44:41 smithi183 ceph-mon[25285]: osdmap e39: 6 total, 6 up, 6 in 2024-03-20T08:44:41.606 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:44:41 smithi183 ceph-mon[31860]: purged_snaps scrub starts 2024-03-20T08:44:41.606 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:44:41 smithi183 ceph-mon[31860]: purged_snaps scrub ok 2024-03-20T08:44:41.606 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:44:41 smithi183 ceph-mon[31860]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-03-20T08:44:41.606 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:44:41 smithi183 ceph-mon[31860]: osdmap e39: 6 total, 6 up, 6 in 2024-03-20T08:44:42.176 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4c3f3eba-e695-11ee-95c9-87774f69a715 -- ceph orch daemon add osd smithi194:/dev/nvme2n1 2024-03-20T08:44:42.416 INFO:teuthology.orchestra.run.smithi194.stderr:Inferring config /var/lib/ceph/4c3f3eba-e695-11ee-95c9-87774f69a715/mon.b/config 2024-03-20T08:44:43.294 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:44:43 smithi194 ceph-mon[27237]: osdmap e40: 6 total, 6 up, 6 in 2024-03-20T08:44:43.295 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:44:43 smithi194 ceph-mon[27237]: pgmap v150: 1 pgs: 1 remapped+peering; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-03-20T08:44:43.454 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:44:43 smithi183 ceph-mon[25285]: osdmap e40: 6 total, 6 up, 6 in 2024-03-20T08:44:43.454 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:44:43 smithi183 ceph-mon[25285]: pgmap v150: 1 pgs: 1 remapped+peering; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-03-20T08:44:43.454 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:44:43 smithi183 ceph-mon[31860]: osdmap e40: 6 total, 6 up, 6 in 2024-03-20T08:44:43.455 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:44:43 smithi183 ceph-mon[31860]: pgmap v150: 1 pgs: 1 remapped+peering; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-03-20T08:44:44.454 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:44:44 smithi183 ceph-mon[25285]: Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY) 2024-03-20T08:44:44.454 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:44:44 smithi183 ceph-mon[25285]: osdmap e41: 6 total, 6 up, 6 in 2024-03-20T08:44:44.455 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:44:44 smithi183 ceph-mon[25285]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-20T08:44:44.455 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:44:44 smithi183 ceph-mon[25285]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-20T08:44:44.455 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:44:44 smithi183 ceph-mon[25285]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-20T08:44:44.455 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:44:44 smithi183 ceph-mon[31860]: Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY) 2024-03-20T08:44:44.455 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:44:44 smithi183 ceph-mon[31860]: osdmap e41: 6 total, 6 up, 6 in 2024-03-20T08:44:44.455 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:44:44 smithi183 ceph-mon[31860]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-20T08:44:44.455 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:44:44 smithi183 ceph-mon[31860]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-20T08:44:44.455 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:44:44 smithi183 ceph-mon[31860]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-20T08:44:44.608 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:44:44 smithi194 ceph-mon[27237]: Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY) 2024-03-20T08:44:44.609 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:44:44 smithi194 ceph-mon[27237]: osdmap e41: 6 total, 6 up, 6 in 2024-03-20T08:44:44.609 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:44:44 smithi194 ceph-mon[27237]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-20T08:44:44.609 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:44:44 smithi194 ceph-mon[27237]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-20T08:44:44.609 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:44:44 smithi194 ceph-mon[27237]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-20T08:44:45.562 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:44:45 smithi194 ceph-mon[27237]: from='client.24250 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi194:/dev/nvme2n1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-20T08:44:45.563 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:44:45 smithi194 ceph-mon[27237]: osdmap e42: 6 total, 6 up, 6 in 2024-03-20T08:44:45.563 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:44:45 smithi194 ceph-mon[27237]: pgmap v153: 1 pgs: 1 remapped+peering; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-03-20T08:44:45.704 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:44:45 smithi183 ceph-mon[25285]: from='client.24250 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi194:/dev/nvme2n1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-20T08:44:45.704 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:44:45 smithi183 ceph-mon[25285]: osdmap e42: 6 total, 6 up, 6 in 2024-03-20T08:44:45.704 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:44:45 smithi183 ceph-mon[25285]: pgmap v153: 1 pgs: 1 remapped+peering; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-03-20T08:44:45.705 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:44:45 smithi183 ceph-mon[31860]: from='client.24250 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi194:/dev/nvme2n1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-20T08:44:45.705 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:44:45 smithi183 ceph-mon[31860]: osdmap e42: 6 total, 6 up, 6 in 2024-03-20T08:44:45.705 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:44:45 smithi183 ceph-mon[31860]: pgmap v153: 1 pgs: 1 remapped+peering; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-03-20T08:44:46.638 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:44:46 smithi194 ceph-mon[27237]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg peering) 2024-03-20T08:44:46.639 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:44:46 smithi194 ceph-mon[27237]: Cluster is now healthy 2024-03-20T08:44:46.704 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:44:46 smithi183 ceph-mon[25285]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg peering) 2024-03-20T08:44:46.704 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:44:46 smithi183 ceph-mon[25285]: Cluster is now healthy 2024-03-20T08:44:46.704 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:44:46 smithi183 ceph-mon[31860]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg peering) 2024-03-20T08:44:46.705 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:44:46 smithi183 ceph-mon[31860]: Cluster is now healthy 2024-03-20T08:44:47.639 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:44:47 smithi194 ceph-mon[27237]: pgmap v154: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-03-20T08:44:47.639 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:44:47 smithi194 ceph-mon[27237]: from='client.? 172.21.15.194:0/1173758290' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "44adcd7a-6dbf-4437-911f-b41c87190193"}]: dispatch 2024-03-20T08:44:47.639 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:44:47 smithi194 ceph-mon[27237]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "44adcd7a-6dbf-4437-911f-b41c87190193"}]: dispatch 2024-03-20T08:44:47.639 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:44:47 smithi194 ceph-mon[27237]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "44adcd7a-6dbf-4437-911f-b41c87190193"}]': finished 2024-03-20T08:44:47.639 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:44:47 smithi194 ceph-mon[27237]: osdmap e43: 7 total, 6 up, 7 in 2024-03-20T08:44:47.639 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:44:47 smithi194 ceph-mon[27237]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-20T08:44:47.704 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:44:47 smithi183 ceph-mon[25285]: pgmap v154: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-03-20T08:44:47.704 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:44:47 smithi183 ceph-mon[25285]: from='client.? 172.21.15.194:0/1173758290' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "44adcd7a-6dbf-4437-911f-b41c87190193"}]: dispatch 2024-03-20T08:44:47.705 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:44:47 smithi183 ceph-mon[25285]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "44adcd7a-6dbf-4437-911f-b41c87190193"}]: dispatch 2024-03-20T08:44:47.705 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:44:47 smithi183 ceph-mon[25285]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "44adcd7a-6dbf-4437-911f-b41c87190193"}]': finished 2024-03-20T08:44:47.705 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:44:47 smithi183 ceph-mon[25285]: osdmap e43: 7 total, 6 up, 7 in 2024-03-20T08:44:47.705 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:44:47 smithi183 ceph-mon[25285]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-20T08:44:47.705 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:44:47 smithi183 ceph-mon[31860]: pgmap v154: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-03-20T08:44:47.705 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:44:47 smithi183 ceph-mon[31860]: from='client.? 172.21.15.194:0/1173758290' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "44adcd7a-6dbf-4437-911f-b41c87190193"}]: dispatch 2024-03-20T08:44:47.705 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:44:47 smithi183 ceph-mon[31860]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "44adcd7a-6dbf-4437-911f-b41c87190193"}]: dispatch 2024-03-20T08:44:47.705 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:44:47 smithi183 ceph-mon[31860]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "44adcd7a-6dbf-4437-911f-b41c87190193"}]': finished 2024-03-20T08:44:47.705 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:44:47 smithi183 ceph-mon[31860]: osdmap e43: 7 total, 6 up, 7 in 2024-03-20T08:44:47.705 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:44:47 smithi183 ceph-mon[31860]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-20T08:44:48.638 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:44:48 smithi194 ceph-mon[27237]: from='client.? 172.21.15.194:0/2706225934' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-20T08:44:48.704 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:44:48 smithi183 ceph-mon[25285]: from='client.? 172.21.15.194:0/2706225934' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-20T08:44:48.705 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:44:48 smithi183 ceph-mon[31860]: from='client.? 172.21.15.194:0/2706225934' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-20T08:44:49.639 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:44:49 smithi194 ceph-mon[27237]: pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-03-20T08:44:49.704 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:44:49 smithi183 ceph-mon[25285]: pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-03-20T08:44:49.704 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:44:49 smithi183 ceph-mon[31860]: pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-03-20T08:44:50.639 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:44:50 smithi194 ceph-mon[27237]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:44:50.639 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:44:50 smithi194 ceph-mon[27237]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:44:50.639 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:44:50 smithi194 ceph-mon[27237]: Detected new or changed devices on smithi194 2024-03-20T08:44:50.639 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:44:50 smithi194 ceph-mon[27237]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:44:50.704 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:44:50 smithi183 ceph-mon[25285]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:44:50.704 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:44:50 smithi183 ceph-mon[25285]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:44:50.704 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:44:50 smithi183 ceph-mon[25285]: Detected new or changed devices on smithi194 2024-03-20T08:44:50.705 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:44:50 smithi183 ceph-mon[25285]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:44:50.705 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:44:50 smithi183 ceph-mon[31860]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:44:50.705 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:44:50 smithi183 ceph-mon[31860]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:44:50.705 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:44:50 smithi183 ceph-mon[31860]: Detected new or changed devices on smithi194 2024-03-20T08:44:50.705 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:44:50 smithi183 ceph-mon[31860]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:44:51.660 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:44:51 smithi183 ceph-mon[25285]: pgmap v157: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail; 82 KiB/s, 0 objects/s recovering 2024-03-20T08:44:51.660 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:44:51 smithi183 ceph-mon[31860]: pgmap v157: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail; 82 KiB/s, 0 objects/s recovering 2024-03-20T08:44:51.661 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:44:51 smithi194 ceph-mon[27237]: pgmap v157: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail; 82 KiB/s, 0 objects/s recovering 2024-03-20T08:44:53.595 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:44:53 smithi194 ceph-mon[27237]: pgmap v158: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-03-20T08:44:53.704 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:44:53 smithi183 ceph-mon[31860]: pgmap v158: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-03-20T08:44:53.704 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:44:53 smithi183 ceph-mon[25285]: pgmap v158: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-03-20T08:44:55.454 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:44:55 smithi183 ceph-mon[25285]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-03-20T08:44:55.455 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:44:55 smithi183 ceph-mon[25285]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-03-20T08:44:55.455 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:44:55 smithi183 ceph-mon[25285]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-20T08:44:55.455 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:44:55 smithi183 ceph-mon[25285]: Deploying daemon osd.6 on smithi194 2024-03-20T08:44:55.455 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:44:55 smithi183 ceph-mon[31860]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-03-20T08:44:55.455 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:44:55 smithi183 ceph-mon[31860]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-03-20T08:44:55.455 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:44:55 smithi183 ceph-mon[31860]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-20T08:44:55.455 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:44:55 smithi183 ceph-mon[31860]: Deploying daemon osd.6 on smithi194 2024-03-20T08:44:55.888 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:44:55 smithi194 ceph-mon[27237]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-03-20T08:44:55.889 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:44:55 smithi194 ceph-mon[27237]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-03-20T08:44:55.889 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:44:55 smithi194 ceph-mon[27237]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-20T08:44:55.889 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:44:55 smithi194 ceph-mon[27237]: Deploying daemon osd.6 on smithi194 2024-03-20T08:44:57.422 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:44:57 smithi183 ceph-mon[25285]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-03-20T08:44:57.422 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:44:57 smithi183 ceph-mon[25285]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:44:57.422 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:44:57 smithi183 ceph-mon[25285]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:44:57.422 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:44:57 smithi183 ceph-mon[25285]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:44:57.422 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:44:57 smithi183 ceph-mon[31860]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-03-20T08:44:57.422 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:44:57 smithi183 ceph-mon[31860]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:44:57.423 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:44:57 smithi183 ceph-mon[31860]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:44:57.423 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:44:57 smithi183 ceph-mon[31860]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:44:57.451 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:44:57 smithi194 ceph-mon[27237]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-03-20T08:44:57.452 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:44:57 smithi194 ceph-mon[27237]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:44:57.452 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:44:57 smithi194 ceph-mon[27237]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:44:57.452 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:44:57 smithi194 ceph-mon[27237]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:44:59.384 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:44:59 smithi194 ceph-mon[27237]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-03-20T08:44:59.447 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:44:59 smithi183 ceph-mon[25285]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-03-20T08:44:59.448 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:44:59 smithi183 ceph-mon[31860]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-03-20T08:45:01.638 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:45:01 smithi194 ceph-mon[27237]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-03-20T08:45:01.639 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:45:01 smithi194 ceph-mon[27237]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:45:01.639 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:45:01 smithi194 ceph-mon[27237]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:45:01.639 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:45:01 smithi194 ceph-mon[27237]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-20T08:45:01.639 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:45:01 smithi194 ceph-mon[27237]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-20T08:45:01.639 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:45:01 smithi194 ceph-mon[27237]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-20T08:45:01.639 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:45:01 smithi194 ceph-mon[27237]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:45:01.704 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:45:01 smithi183 ceph-mon[31860]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-03-20T08:45:01.704 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:45:01 smithi183 ceph-mon[31860]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:45:01.704 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:45:01 smithi183 ceph-mon[31860]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:45:01.704 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:45:01 smithi183 ceph-mon[31860]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-20T08:45:01.704 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:45:01 smithi183 ceph-mon[31860]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-20T08:45:01.705 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:45:01 smithi183 ceph-mon[31860]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-20T08:45:01.705 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:45:01 smithi183 ceph-mon[31860]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:45:01.705 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:45:01 smithi183 ceph-mon[25285]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-03-20T08:45:01.705 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:45:01 smithi183 ceph-mon[25285]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:45:01.705 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:45:01 smithi183 ceph-mon[25285]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:45:01.705 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:45:01 smithi183 ceph-mon[25285]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-20T08:45:01.705 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:45:01 smithi183 ceph-mon[25285]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-20T08:45:01.705 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:45:01 smithi183 ceph-mon[25285]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-20T08:45:01.705 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:45:01 smithi183 ceph-mon[25285]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:45:03.639 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:45:03 smithi194 ceph-mon[27237]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-03-20T08:45:03.704 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:45:03 smithi183 ceph-mon[31860]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-03-20T08:45:03.705 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:45:03 smithi183 ceph-mon[25285]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-03-20T08:45:04.074 INFO:teuthology.orchestra.run.smithi194.stdout:Created osd(s) 6 on host 'smithi194' 2024-03-20T08:45:04.657 DEBUG:teuthology.orchestra.run.smithi194:osd.6> sudo journalctl -f -n 0 -u ceph-4c3f3eba-e695-11ee-95c9-87774f69a715@osd.6.service 2024-03-20T08:45:04.660 INFO:tasks.cephadm:Deploying osd.7 on smithi194 with /dev/nvme1n1... 2024-03-20T08:45:04.660 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4c3f3eba-e695-11ee-95c9-87774f69a715 -- ceph-volume lvm zap /dev/nvme1n1 2024-03-20T08:45:04.933 INFO:teuthology.orchestra.run.smithi194.stderr:Inferring config /var/lib/ceph/4c3f3eba-e695-11ee-95c9-87774f69a715/mon.b/config 2024-03-20T08:45:05.389 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:45:05 smithi194 ceph-mon[27237]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:45:05.389 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:45:05 smithi194 ceph-mon[27237]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:45:05.389 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:45:05 smithi194 ceph-mon[27237]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-20T08:45:05.389 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:45:05 smithi194 ceph-mon[27237]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-20T08:45:05.389 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:45:05 smithi194 ceph-mon[27237]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-20T08:45:05.389 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:45:05 smithi194 ceph-mon[27237]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-20T08:45:05.389 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:45:05 smithi194 ceph-mon[27237]: Adjusting osd_memory_target on smithi194 to 5354M 2024-03-20T08:45:05.389 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:45:05 smithi194 ceph-mon[27237]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:45:05.390 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:45:05 smithi194 ceph-mon[27237]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-20T08:45:05.390 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:45:05 smithi194 ceph-mon[27237]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-20T08:45:05.390 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:45:05 smithi194 ceph-mon[27237]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:45:05.390 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:45:05 smithi194 ceph-mon[27237]: from='osd.6 [v2:172.21.15.194:6816/3187912321,v1:172.21.15.194:6817/3187912321]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-03-20T08:45:05.390 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:45:05 smithi194 ceph-mon[27237]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-03-20T08:45:05.390 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:45:05 smithi194 ceph-mon[27237]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-03-20T08:45:05.454 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:45:05 smithi183 ceph-mon[31860]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:45:05.455 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:45:05 smithi183 ceph-mon[31860]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:45:05.455 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:45:05 smithi183 ceph-mon[31860]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-20T08:45:05.455 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:45:05 smithi183 ceph-mon[31860]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-20T08:45:05.455 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:45:05 smithi183 ceph-mon[31860]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-20T08:45:05.455 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:45:05 smithi183 ceph-mon[31860]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-20T08:45:05.455 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:45:05 smithi183 ceph-mon[31860]: Adjusting osd_memory_target on smithi194 to 5354M 2024-03-20T08:45:05.455 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:45:05 smithi183 ceph-mon[31860]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:45:05.455 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:45:05 smithi183 ceph-mon[31860]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-20T08:45:05.455 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:45:05 smithi183 ceph-mon[31860]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-20T08:45:05.455 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:45:05 smithi183 ceph-mon[31860]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:45:05.455 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:45:05 smithi183 ceph-mon[31860]: from='osd.6 [v2:172.21.15.194:6816/3187912321,v1:172.21.15.194:6817/3187912321]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-03-20T08:45:05.455 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:45:05 smithi183 ceph-mon[31860]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-03-20T08:45:05.455 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:45:05 smithi183 ceph-mon[31860]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-03-20T08:45:05.456 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:45:05 smithi183 ceph-mon[25285]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:45:05.456 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:45:05 smithi183 ceph-mon[25285]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:45:05.456 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:45:05 smithi183 ceph-mon[25285]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-20T08:45:05.456 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:45:05 smithi183 ceph-mon[25285]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-20T08:45:05.456 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:45:05 smithi183 ceph-mon[25285]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-20T08:45:05.456 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:45:05 smithi183 ceph-mon[25285]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-20T08:45:05.456 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:45:05 smithi183 ceph-mon[25285]: Adjusting osd_memory_target on smithi194 to 5354M 2024-03-20T08:45:05.456 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:45:05 smithi183 ceph-mon[25285]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:45:05.456 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:45:05 smithi183 ceph-mon[25285]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-20T08:45:05.456 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:45:05 smithi183 ceph-mon[25285]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-20T08:45:05.456 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:45:05 smithi183 ceph-mon[25285]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:45:05.456 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:45:05 smithi183 ceph-mon[25285]: from='osd.6 [v2:172.21.15.194:6816/3187912321,v1:172.21.15.194:6817/3187912321]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-03-20T08:45:05.457 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:45:05 smithi183 ceph-mon[25285]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-03-20T08:45:05.457 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:45:05 smithi183 ceph-mon[25285]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-03-20T08:45:06.340 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:45:06 smithi194 ceph-mon[27237]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-03-20T08:45:06.340 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:45:06 smithi194 ceph-mon[27237]: osdmap e44: 7 total, 6 up, 7 in 2024-03-20T08:45:06.340 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:45:06 smithi194 ceph-mon[27237]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-20T08:45:06.340 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:45:06 smithi194 ceph-mon[27237]: from='osd.6 [v2:172.21.15.194:6816/3187912321,v1:172.21.15.194:6817/3187912321]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi194", "root=default"]}]: dispatch 2024-03-20T08:45:06.340 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:45:06 smithi194 ceph-mon[27237]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi194", "root=default"]}]: dispatch 2024-03-20T08:45:06.454 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:45:06 smithi183 ceph-mon[31860]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-03-20T08:45:06.454 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:45:06 smithi183 ceph-mon[31860]: osdmap e44: 7 total, 6 up, 7 in 2024-03-20T08:45:06.455 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:45:06 smithi183 ceph-mon[31860]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-20T08:45:06.455 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:45:06 smithi183 ceph-mon[31860]: from='osd.6 [v2:172.21.15.194:6816/3187912321,v1:172.21.15.194:6817/3187912321]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi194", "root=default"]}]: dispatch 2024-03-20T08:45:06.455 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:45:06 smithi183 ceph-mon[31860]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi194", "root=default"]}]: dispatch 2024-03-20T08:45:06.455 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:45:06 smithi183 ceph-mon[25285]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-03-20T08:45:06.455 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:45:06 smithi183 ceph-mon[25285]: osdmap e44: 7 total, 6 up, 7 in 2024-03-20T08:45:06.455 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:45:06 smithi183 ceph-mon[25285]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-20T08:45:06.455 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:45:06 smithi183 ceph-mon[25285]: from='osd.6 [v2:172.21.15.194:6816/3187912321,v1:172.21.15.194:6817/3187912321]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi194", "root=default"]}]: dispatch 2024-03-20T08:45:06.455 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:45:06 smithi183 ceph-mon[25285]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi194", "root=default"]}]: dispatch 2024-03-20T08:45:06.639 INFO:journalctl@ceph.osd.6.smithi194.stdout:Mar 20 08:45:06 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-osd-6[41285]: 2024-03-20T08:45:06.560+0000 7fabbb776700 -1 osd.6 0 waiting for initial osdmap 2024-03-20T08:45:06.639 INFO:journalctl@ceph.osd.6.smithi194.stdout:Mar 20 08:45:06 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-osd-6[41285]: 2024-03-20T08:45:06.566+0000 7fabb3561700 -1 osd.6 45 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-20T08:45:06.728 INFO:teuthology.orchestra.run.smithi194.stderr:--> Zapping: /dev/nvme1n1 2024-03-20T08:45:06.728 INFO:teuthology.orchestra.run.smithi194.stderr:--> --destroy was not specified, but zapping a whole device will remove the partition table 2024-03-20T08:45:06.728 INFO:teuthology.orchestra.run.smithi194.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/nvme1n1 bs=1M count=10 conv=fsync 2024-03-20T08:45:06.728 INFO:teuthology.orchestra.run.smithi194.stderr: stderr: 10+0 records in 2024-03-20T08:45:06.728 INFO:teuthology.orchestra.run.smithi194.stderr:10+0 records out 2024-03-20T08:45:06.728 INFO:teuthology.orchestra.run.smithi194.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0132611 s, 791 MB/s 2024-03-20T08:45:06.728 INFO:teuthology.orchestra.run.smithi194.stderr:--> Zapping successful for: 2024-03-20T08:45:07.283 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4c3f3eba-e695-11ee-95c9-87774f69a715 -- ceph orch daemon add osd smithi194:/dev/nvme1n1 2024-03-20T08:45:07.388 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:45:07 smithi194 ceph-mon[27237]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi194", "root=default"]}]': finished 2024-03-20T08:45:07.389 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:45:07 smithi194 ceph-mon[27237]: osdmap e45: 7 total, 6 up, 7 in 2024-03-20T08:45:07.389 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:45:07 smithi194 ceph-mon[27237]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-20T08:45:07.389 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:45:07 smithi194 ceph-mon[27237]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-20T08:45:07.389 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:45:07 smithi194 ceph-mon[27237]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-03-20T08:45:07.389 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:45:07 smithi194 ceph-mon[27237]: from='osd.6 ' entity='osd.6' 2024-03-20T08:45:07.389 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:45:07 smithi194 ceph-mon[27237]: osd.6 [v2:172.21.15.194:6816/3187912321,v1:172.21.15.194:6817/3187912321] boot 2024-03-20T08:45:07.389 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:45:07 smithi194 ceph-mon[27237]: osdmap e46: 7 total, 7 up, 7 in 2024-03-20T08:45:07.389 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:45:07 smithi194 ceph-mon[27237]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-20T08:45:07.454 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:45:07 smithi183 ceph-mon[31860]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi194", "root=default"]}]': finished 2024-03-20T08:45:07.455 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:45:07 smithi183 ceph-mon[31860]: osdmap e45: 7 total, 6 up, 7 in 2024-03-20T08:45:07.455 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:45:07 smithi183 ceph-mon[31860]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-20T08:45:07.455 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:45:07 smithi183 ceph-mon[31860]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-20T08:45:07.455 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:45:07 smithi183 ceph-mon[31860]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-03-20T08:45:07.455 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:45:07 smithi183 ceph-mon[31860]: from='osd.6 ' entity='osd.6' 2024-03-20T08:45:07.456 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:45:07 smithi183 ceph-mon[31860]: osd.6 [v2:172.21.15.194:6816/3187912321,v1:172.21.15.194:6817/3187912321] boot 2024-03-20T08:45:07.456 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:45:07 smithi183 ceph-mon[31860]: osdmap e46: 7 total, 7 up, 7 in 2024-03-20T08:45:07.456 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:45:07 smithi183 ceph-mon[31860]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-20T08:45:07.456 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:45:07 smithi183 ceph-mon[25285]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi194", "root=default"]}]': finished 2024-03-20T08:45:07.456 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:45:07 smithi183 ceph-mon[25285]: osdmap e45: 7 total, 6 up, 7 in 2024-03-20T08:45:07.457 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:45:07 smithi183 ceph-mon[25285]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-20T08:45:07.457 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:45:07 smithi183 ceph-mon[25285]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-20T08:45:07.457 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:45:07 smithi183 ceph-mon[25285]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-03-20T08:45:07.457 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:45:07 smithi183 ceph-mon[25285]: from='osd.6 ' entity='osd.6' 2024-03-20T08:45:07.457 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:45:07 smithi183 ceph-mon[25285]: osd.6 [v2:172.21.15.194:6816/3187912321,v1:172.21.15.194:6817/3187912321] boot 2024-03-20T08:45:07.457 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:45:07 smithi183 ceph-mon[25285]: osdmap e46: 7 total, 7 up, 7 in 2024-03-20T08:45:07.457 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:45:07 smithi183 ceph-mon[25285]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-20T08:45:07.574 INFO:teuthology.orchestra.run.smithi194.stderr:Inferring config /var/lib/ceph/4c3f3eba-e695-11ee-95c9-87774f69a715/mon.b/config 2024-03-20T08:45:08.390 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:45:08 smithi194 ceph-mon[27237]: purged_snaps scrub starts 2024-03-20T08:45:08.390 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:45:08 smithi194 ceph-mon[27237]: purged_snaps scrub ok 2024-03-20T08:45:08.390 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:45:08 smithi194 ceph-mon[27237]: osdmap e47: 7 total, 7 up, 7 in 2024-03-20T08:45:08.454 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:45:08 smithi183 ceph-mon[25285]: purged_snaps scrub starts 2024-03-20T08:45:08.454 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:45:08 smithi183 ceph-mon[25285]: purged_snaps scrub ok 2024-03-20T08:45:08.454 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:45:08 smithi183 ceph-mon[25285]: osdmap e47: 7 total, 7 up, 7 in 2024-03-20T08:45:08.455 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:45:08 smithi183 ceph-mon[31860]: purged_snaps scrub starts 2024-03-20T08:45:08.455 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:45:08 smithi183 ceph-mon[31860]: purged_snaps scrub ok 2024-03-20T08:45:08.455 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:45:08 smithi183 ceph-mon[31860]: osdmap e47: 7 total, 7 up, 7 in 2024-03-20T08:45:09.329 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:45:09 smithi194 ceph-mon[27237]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 381 MiB used, 625 GiB / 626 GiB avail 2024-03-20T08:45:09.454 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:45:09 smithi183 ceph-mon[25285]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 381 MiB used, 625 GiB / 626 GiB avail 2024-03-20T08:45:09.455 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:45:09 smithi183 ceph-mon[31860]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 381 MiB used, 625 GiB / 626 GiB avail 2024-03-20T08:45:10.235 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:45:10 smithi194 ceph-mon[27237]: osdmap e48: 7 total, 7 up, 7 in 2024-03-20T08:45:10.235 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:45:10 smithi194 ceph-mon[27237]: from='client.24277 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi194:/dev/nvme1n1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-20T08:45:10.235 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:45:10 smithi194 ceph-mon[27237]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-20T08:45:10.235 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:45:10 smithi194 ceph-mon[27237]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-20T08:45:10.235 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:45:10 smithi194 ceph-mon[27237]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-20T08:45:10.236 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:45:10 smithi194 ceph-mon[27237]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:45:10.236 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:45:10 smithi194 ceph-mon[27237]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:45:10.236 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:45:10 smithi194 ceph-mon[27237]: Detected new or changed devices on smithi194 2024-03-20T08:45:10.236 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:45:10 smithi194 ceph-mon[27237]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:45:10.236 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:45:10 smithi194 ceph-mon[27237]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-20T08:45:10.236 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:45:10 smithi194 ceph-mon[27237]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-20T08:45:10.236 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:45:10 smithi194 ceph-mon[27237]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-20T08:45:10.236 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:45:10 smithi194 ceph-mon[27237]: pgmap v172: 1 pgs: 1 active+clean; 577 KiB data, 381 MiB used, 625 GiB / 626 GiB avail 2024-03-20T08:45:10.236 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:45:10 smithi194 ceph-mon[27237]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:45:10.454 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:45:10 smithi183 ceph-mon[25285]: osdmap e48: 7 total, 7 up, 7 in 2024-03-20T08:45:10.454 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:45:10 smithi183 ceph-mon[25285]: from='client.24277 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi194:/dev/nvme1n1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-20T08:45:10.454 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:45:10 smithi183 ceph-mon[25285]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-20T08:45:10.455 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:45:10 smithi183 ceph-mon[25285]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-20T08:45:10.455 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:45:10 smithi183 ceph-mon[25285]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-20T08:45:10.455 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:45:10 smithi183 ceph-mon[25285]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:45:10.455 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:45:10 smithi183 ceph-mon[25285]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:45:10.455 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:45:10 smithi183 ceph-mon[25285]: Detected new or changed devices on smithi194 2024-03-20T08:45:10.455 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:45:10 smithi183 ceph-mon[25285]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:45:10.455 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:45:10 smithi183 ceph-mon[25285]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-20T08:45:10.455 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:45:10 smithi183 ceph-mon[25285]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-20T08:45:10.455 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:45:10 smithi183 ceph-mon[25285]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-20T08:45:10.455 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:45:10 smithi183 ceph-mon[25285]: pgmap v172: 1 pgs: 1 active+clean; 577 KiB data, 381 MiB used, 625 GiB / 626 GiB avail 2024-03-20T08:45:10.455 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:45:10 smithi183 ceph-mon[25285]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:45:10.455 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:45:10 smithi183 ceph-mon[31860]: osdmap e48: 7 total, 7 up, 7 in 2024-03-20T08:45:10.456 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:45:10 smithi183 ceph-mon[31860]: from='client.24277 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi194:/dev/nvme1n1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-20T08:45:10.456 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:45:10 smithi183 ceph-mon[31860]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-20T08:45:10.456 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:45:10 smithi183 ceph-mon[31860]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-20T08:45:10.456 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:45:10 smithi183 ceph-mon[31860]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-20T08:45:10.456 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:45:10 smithi183 ceph-mon[31860]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:45:10.456 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:45:10 smithi183 ceph-mon[31860]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:45:10.456 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:45:10 smithi183 ceph-mon[31860]: Detected new or changed devices on smithi194 2024-03-20T08:45:10.456 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:45:10 smithi183 ceph-mon[31860]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:45:10.456 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:45:10 smithi183 ceph-mon[31860]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-20T08:45:10.456 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:45:10 smithi183 ceph-mon[31860]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-20T08:45:10.456 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:45:10 smithi183 ceph-mon[31860]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-20T08:45:10.456 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:45:10 smithi183 ceph-mon[31860]: pgmap v172: 1 pgs: 1 active+clean; 577 KiB data, 381 MiB used, 625 GiB / 626 GiB avail 2024-03-20T08:45:10.456 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:45:10 smithi183 ceph-mon[31860]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:45:11.368 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:45:11 smithi183 ceph-mon[31860]: osdmap e49: 7 total, 7 up, 7 in 2024-03-20T08:45:11.368 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:45:11 smithi183 ceph-mon[31860]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:45:11.368 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:45:11 smithi183 ceph-mon[31860]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:45:11.368 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:45:11 smithi183 ceph-mon[31860]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:45:11.368 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:45:11 smithi183 ceph-mon[31860]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-20T08:45:11.368 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:45:11 smithi183 ceph-mon[31860]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-20T08:45:11.368 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:45:11 smithi183 ceph-mon[31860]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-20T08:45:11.368 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:45:11 smithi183 ceph-mon[31860]: pgmap v174: 1 pgs: 1 active+clean; 577 KiB data, 381 MiB used, 625 GiB / 626 GiB avail 2024-03-20T08:45:11.368 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:45:11 smithi183 ceph-mon[31860]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:45:11.368 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:45:11 smithi183 ceph-mon[25285]: osdmap e49: 7 total, 7 up, 7 in 2024-03-20T08:45:11.369 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:45:11 smithi183 ceph-mon[25285]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:45:11.369 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:45:11 smithi183 ceph-mon[25285]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:45:11.369 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:45:11 smithi183 ceph-mon[25285]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:45:11.369 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:45:11 smithi183 ceph-mon[25285]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-20T08:45:11.369 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:45:11 smithi183 ceph-mon[25285]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-20T08:45:11.369 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:45:11 smithi183 ceph-mon[25285]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-20T08:45:11.369 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:45:11 smithi183 ceph-mon[25285]: pgmap v174: 1 pgs: 1 active+clean; 577 KiB data, 381 MiB used, 625 GiB / 626 GiB avail 2024-03-20T08:45:11.369 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:45:11 smithi183 ceph-mon[25285]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:45:11.389 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:45:11 smithi194 ceph-mon[27237]: osdmap e49: 7 total, 7 up, 7 in 2024-03-20T08:45:11.389 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:45:11 smithi194 ceph-mon[27237]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:45:11.389 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:45:11 smithi194 ceph-mon[27237]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:45:11.389 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:45:11 smithi194 ceph-mon[27237]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:45:11.389 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:45:11 smithi194 ceph-mon[27237]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-20T08:45:11.389 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:45:11 smithi194 ceph-mon[27237]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-20T08:45:11.389 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:45:11 smithi194 ceph-mon[27237]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-20T08:45:11.389 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:45:11 smithi194 ceph-mon[27237]: pgmap v174: 1 pgs: 1 active+clean; 577 KiB data, 381 MiB used, 625 GiB / 626 GiB avail 2024-03-20T08:45:11.389 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:45:11 smithi194 ceph-mon[27237]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:45:12.339 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:45:12 smithi194 ceph-mon[27237]: osdmap e50: 7 total, 7 up, 7 in 2024-03-20T08:45:12.339 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:45:12 smithi194 ceph-mon[27237]: from='client.? 172.21.15.194:0/3713312864' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "87e8f364-5518-446c-84d8-7443f76a3bd9"}]: dispatch 2024-03-20T08:45:12.339 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:45:12 smithi194 ceph-mon[27237]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "87e8f364-5518-446c-84d8-7443f76a3bd9"}]: dispatch 2024-03-20T08:45:12.339 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:45:12 smithi194 ceph-mon[27237]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "87e8f364-5518-446c-84d8-7443f76a3bd9"}]': finished 2024-03-20T08:45:12.339 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:45:12 smithi194 ceph-mon[27237]: osdmap e51: 8 total, 7 up, 8 in 2024-03-20T08:45:12.339 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:45:12 smithi194 ceph-mon[27237]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-20T08:45:12.454 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:45:12 smithi183 ceph-mon[31860]: osdmap e50: 7 total, 7 up, 7 in 2024-03-20T08:45:12.454 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:45:12 smithi183 ceph-mon[31860]: from='client.? 172.21.15.194:0/3713312864' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "87e8f364-5518-446c-84d8-7443f76a3bd9"}]: dispatch 2024-03-20T08:45:12.454 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:45:12 smithi183 ceph-mon[31860]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "87e8f364-5518-446c-84d8-7443f76a3bd9"}]: dispatch 2024-03-20T08:45:12.455 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:45:12 smithi183 ceph-mon[31860]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "87e8f364-5518-446c-84d8-7443f76a3bd9"}]': finished 2024-03-20T08:45:12.455 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:45:12 smithi183 ceph-mon[31860]: osdmap e51: 8 total, 7 up, 8 in 2024-03-20T08:45:12.455 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:45:12 smithi183 ceph-mon[31860]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-20T08:45:12.455 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:45:12 smithi183 ceph-mon[25285]: osdmap e50: 7 total, 7 up, 7 in 2024-03-20T08:45:12.455 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:45:12 smithi183 ceph-mon[25285]: from='client.? 172.21.15.194:0/3713312864' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "87e8f364-5518-446c-84d8-7443f76a3bd9"}]: dispatch 2024-03-20T08:45:12.455 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:45:12 smithi183 ceph-mon[25285]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "87e8f364-5518-446c-84d8-7443f76a3bd9"}]: dispatch 2024-03-20T08:45:12.455 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:45:12 smithi183 ceph-mon[25285]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "87e8f364-5518-446c-84d8-7443f76a3bd9"}]': finished 2024-03-20T08:45:12.455 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:45:12 smithi183 ceph-mon[25285]: osdmap e51: 8 total, 7 up, 8 in 2024-03-20T08:45:12.455 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:45:12 smithi183 ceph-mon[25285]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-20T08:45:13.388 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:45:13 smithi194 ceph-mon[27237]: pgmap v177: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail; 182 KiB/s, 0 objects/s recovering 2024-03-20T08:45:13.454 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:45:13 smithi183 ceph-mon[25285]: pgmap v177: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail; 182 KiB/s, 0 objects/s recovering 2024-03-20T08:45:13.454 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:45:13 smithi183 ceph-mon[31860]: pgmap v177: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail; 182 KiB/s, 0 objects/s recovering 2024-03-20T08:45:14.388 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:45:14 smithi194 ceph-mon[27237]: from='client.? 172.21.15.194:0/1484673441' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-20T08:45:14.455 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:45:14 smithi183 ceph-mon[25285]: from='client.? 172.21.15.194:0/1484673441' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-20T08:45:14.455 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:45:14 smithi183 ceph-mon[31860]: from='client.? 172.21.15.194:0/1484673441' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-20T08:45:15.388 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:45:15 smithi194 ceph-mon[27237]: pgmap v178: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail; 123 KiB/s, 0 objects/s recovering 2024-03-20T08:45:15.454 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:45:15 smithi183 ceph-mon[25285]: pgmap v178: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail; 123 KiB/s, 0 objects/s recovering 2024-03-20T08:45:15.454 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:45:15 smithi183 ceph-mon[31860]: pgmap v178: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail; 123 KiB/s, 0 objects/s recovering 2024-03-20T08:45:17.478 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:45:17 smithi194 ceph-mon[27237]: pgmap v179: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail; 94 KiB/s, 0 objects/s recovering 2024-03-20T08:45:17.478 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:45:17 smithi194 ceph-mon[27237]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:45:17.478 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:45:17 smithi194 ceph-mon[27237]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:45:17.478 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:45:17 smithi194 ceph-mon[27237]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:45:17.705 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:45:17 smithi183 ceph-mon[25285]: pgmap v179: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail; 94 KiB/s, 0 objects/s recovering 2024-03-20T08:45:17.705 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:45:17 smithi183 ceph-mon[25285]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:45:17.705 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:45:17 smithi183 ceph-mon[25285]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:45:17.705 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:45:17 smithi183 ceph-mon[25285]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:45:17.705 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:45:17 smithi183 ceph-mon[31860]: pgmap v179: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail; 94 KiB/s, 0 objects/s recovering 2024-03-20T08:45:17.705 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:45:17 smithi183 ceph-mon[31860]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:45:17.705 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:45:17 smithi183 ceph-mon[31860]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:45:17.705 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:45:17 smithi183 ceph-mon[31860]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:45:19.683 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:45:19 smithi194 ceph-mon[27237]: pgmap v180: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-03-20T08:45:19.704 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:45:19 smithi183 ceph-mon[25285]: pgmap v180: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-03-20T08:45:19.704 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:45:19 smithi183 ceph-mon[31860]: pgmap v180: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-03-20T08:45:20.602 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:45:20 smithi194 ceph-mon[27237]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-03-20T08:45:20.602 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:45:20 smithi194 ceph-mon[27237]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-20T08:45:20.603 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:45:20 smithi194 ceph-mon[27237]: Deploying daemon osd.7 on smithi194 2024-03-20T08:45:20.652 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:45:20 smithi183 ceph-mon[25285]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-03-20T08:45:20.652 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:45:20 smithi183 ceph-mon[25285]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-20T08:45:20.652 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:45:20 smithi183 ceph-mon[25285]: Deploying daemon osd.7 on smithi194 2024-03-20T08:45:20.652 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:45:20 smithi183 ceph-mon[31860]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-03-20T08:45:20.652 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:45:20 smithi183 ceph-mon[31860]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-20T08:45:20.652 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:45:20 smithi183 ceph-mon[31860]: Deploying daemon osd.7 on smithi194 2024-03-20T08:45:21.639 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:45:21 smithi194 ceph-mon[27237]: pgmap v181: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail; 63 KiB/s, 0 objects/s recovering 2024-03-20T08:45:21.954 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:45:21 smithi183 ceph-mon[25285]: pgmap v181: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail; 63 KiB/s, 0 objects/s recovering 2024-03-20T08:45:21.954 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:45:21 smithi183 ceph-mon[31860]: pgmap v181: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail; 63 KiB/s, 0 objects/s recovering 2024-03-20T08:45:23.890 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:45:23 smithi194 ceph-mon[27237]: pgmap v182: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail; 57 KiB/s, 0 objects/s recovering 2024-03-20T08:45:23.954 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:45:23 smithi183 ceph-mon[25285]: pgmap v182: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail; 57 KiB/s, 0 objects/s recovering 2024-03-20T08:45:23.955 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:45:23 smithi183 ceph-mon[31860]: pgmap v182: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail; 57 KiB/s, 0 objects/s recovering 2024-03-20T08:45:25.781 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:45:25 smithi194 ceph-mon[27237]: pgmap v183: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail 2024-03-20T08:45:25.796 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:45:25 smithi194 ceph-mon[27237]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:45:25.797 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:45:25 smithi194 ceph-mon[27237]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:45:25.797 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:45:25 smithi194 ceph-mon[27237]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-20T08:45:25.797 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:45:25 smithi194 ceph-mon[27237]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-20T08:45:25.797 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:45:25 smithi194 ceph-mon[27237]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-20T08:45:25.797 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:45:25 smithi194 ceph-mon[27237]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:45:25.954 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:45:25 smithi183 ceph-mon[25285]: pgmap v183: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail 2024-03-20T08:45:25.955 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:45:25 smithi183 ceph-mon[25285]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:45:25.955 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:45:25 smithi183 ceph-mon[25285]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:45:25.955 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:45:25 smithi183 ceph-mon[25285]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-20T08:45:25.955 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:45:25 smithi183 ceph-mon[25285]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-20T08:45:25.955 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:45:25 smithi183 ceph-mon[25285]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-20T08:45:25.955 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:45:25 smithi183 ceph-mon[25285]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:45:25.955 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:45:25 smithi183 ceph-mon[31860]: pgmap v183: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail 2024-03-20T08:45:25.955 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:45:25 smithi183 ceph-mon[31860]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:45:25.955 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:45:25 smithi183 ceph-mon[31860]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:45:25.955 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:45:25 smithi183 ceph-mon[31860]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-20T08:45:25.955 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:45:25 smithi183 ceph-mon[31860]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-20T08:45:25.956 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:45:25 smithi183 ceph-mon[31860]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-20T08:45:25.956 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:45:25 smithi183 ceph-mon[31860]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:45:27.731 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:45:27 smithi194 ceph-mon[27237]: pgmap v184: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail 2024-03-20T08:45:27.954 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:45:27 smithi183 ceph-mon[25285]: pgmap v184: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail 2024-03-20T08:45:27.954 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:45:27 smithi183 ceph-mon[31860]: pgmap v184: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail 2024-03-20T08:45:28.337 INFO:teuthology.orchestra.run.smithi194.stdout:Created osd(s) 7 on host 'smithi194' 2024-03-20T08:45:28.639 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:45:28 smithi194 ceph-mon[27237]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-20T08:45:28.639 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:45:28 smithi194 ceph-mon[27237]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:45:28.639 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:45:28 smithi194 ceph-mon[27237]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-20T08:45:28.639 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:45:28 smithi194 ceph-mon[27237]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-20T08:45:28.639 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:45:28 smithi194 ceph-mon[27237]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:45:28.639 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:45:28 smithi194 ceph-mon[27237]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-20T08:45:28.639 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:45:28 smithi194 ceph-mon[27237]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-20T08:45:28.639 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:45:28 smithi194 ceph-mon[27237]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:45:28.639 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:45:28 smithi194 ceph-mon[27237]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-20T08:45:28.639 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:45:28 smithi194 ceph-mon[27237]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-20T08:45:28.639 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:45:28 smithi194 ceph-mon[27237]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:45:28.954 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:45:28 smithi183 ceph-mon[25285]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-20T08:45:28.954 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:45:28 smithi183 ceph-mon[25285]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:45:28.955 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:45:28 smithi183 ceph-mon[25285]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-20T08:45:28.955 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:45:28 smithi183 ceph-mon[25285]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-20T08:45:28.955 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:45:28 smithi183 ceph-mon[25285]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:45:28.955 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:45:28 smithi183 ceph-mon[25285]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-20T08:45:28.955 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:45:28 smithi183 ceph-mon[25285]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-20T08:45:28.955 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:45:28 smithi183 ceph-mon[25285]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:45:28.955 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:45:28 smithi183 ceph-mon[25285]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-20T08:45:28.955 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:45:28 smithi183 ceph-mon[25285]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-20T08:45:28.955 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:45:28 smithi183 ceph-mon[25285]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:45:28.955 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:45:28 smithi183 ceph-mon[31860]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-20T08:45:28.955 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:45:28 smithi183 ceph-mon[31860]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:45:28.955 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:45:28 smithi183 ceph-mon[31860]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-20T08:45:28.956 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:45:28 smithi183 ceph-mon[31860]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-20T08:45:28.956 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:45:28 smithi183 ceph-mon[31860]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:45:28.956 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:45:28 smithi183 ceph-mon[31860]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-20T08:45:28.956 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:45:28 smithi183 ceph-mon[31860]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-20T08:45:28.956 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:45:28 smithi183 ceph-mon[31860]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:45:28.956 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:45:28 smithi183 ceph-mon[31860]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-20T08:45:28.956 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:45:28 smithi183 ceph-mon[31860]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-20T08:45:28.956 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:45:28 smithi183 ceph-mon[31860]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:45:28.959 DEBUG:teuthology.orchestra.run.smithi194:osd.7> sudo journalctl -f -n 0 -u ceph-4c3f3eba-e695-11ee-95c9-87774f69a715@osd.7.service 2024-03-20T08:45:28.961 INFO:tasks.cephadm:Waiting for 8 OSDs to come up... 2024-03-20T08:45:28.961 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4c3f3eba-e695-11ee-95c9-87774f69a715 -- ceph osd stat -f json 2024-03-20T08:45:29.215 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/4c3f3eba-e695-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T08:45:29.663 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:45:29 smithi183 ceph-mon[25285]: pgmap v185: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail 2024-03-20T08:45:29.663 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:45:29 smithi183 ceph-mon[25285]: Adjusting osd_memory_target on smithi194 to 4016M 2024-03-20T08:45:29.663 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:45:29 smithi183 ceph-mon[25285]: from='osd.7 [v2:172.21.15.194:6824/3170101984,v1:172.21.15.194:6825/3170101984]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-20T08:45:29.663 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:45:29 smithi183 ceph-mon[25285]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-20T08:45:29.663 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:45:29 smithi183 ceph-mon[31860]: pgmap v185: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail 2024-03-20T08:45:29.664 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:45:29 smithi183 ceph-mon[31860]: Adjusting osd_memory_target on smithi194 to 4016M 2024-03-20T08:45:29.664 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:45:29 smithi183 ceph-mon[31860]: from='osd.7 [v2:172.21.15.194:6824/3170101984,v1:172.21.15.194:6825/3170101984]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-20T08:45:29.664 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:45:29 smithi183 ceph-mon[31860]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-20T08:45:29.888 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:45:29 smithi194 ceph-mon[27237]: pgmap v185: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail 2024-03-20T08:45:29.889 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:45:29 smithi194 ceph-mon[27237]: Adjusting osd_memory_target on smithi194 to 4016M 2024-03-20T08:45:29.889 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:45:29 smithi194 ceph-mon[27237]: from='osd.7 [v2:172.21.15.194:6824/3170101984,v1:172.21.15.194:6825/3170101984]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-20T08:45:29.889 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:45:29 smithi194 ceph-mon[27237]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-20T08:45:30.639 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:45:30 smithi194 ceph-mon[27237]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-03-20T08:45:30.639 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:45:30 smithi194 ceph-mon[27237]: osdmap e52: 8 total, 7 up, 8 in 2024-03-20T08:45:30.639 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:45:30 smithi194 ceph-mon[27237]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-20T08:45:30.639 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:45:30 smithi194 ceph-mon[27237]: from='osd.7 [v2:172.21.15.194:6824/3170101984,v1:172.21.15.194:6825/3170101984]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi194", "root=default"]}]: dispatch 2024-03-20T08:45:30.639 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:45:30 smithi194 ceph-mon[27237]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi194", "root=default"]}]: dispatch 2024-03-20T08:45:30.639 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:45:30 smithi194 ceph-mon[27237]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:45:30.639 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:45:30 smithi194 ceph-mon[27237]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:45:30.639 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:45:30 smithi194 ceph-mon[27237]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:45:30.704 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:45:30 smithi183 ceph-mon[25285]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-03-20T08:45:30.704 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:45:30 smithi183 ceph-mon[25285]: osdmap e52: 8 total, 7 up, 8 in 2024-03-20T08:45:30.704 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:45:30 smithi183 ceph-mon[25285]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-20T08:45:30.704 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:45:30 smithi183 ceph-mon[25285]: from='osd.7 [v2:172.21.15.194:6824/3170101984,v1:172.21.15.194:6825/3170101984]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi194", "root=default"]}]: dispatch 2024-03-20T08:45:30.704 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:45:30 smithi183 ceph-mon[25285]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi194", "root=default"]}]: dispatch 2024-03-20T08:45:30.705 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:45:30 smithi183 ceph-mon[25285]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:45:30.705 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:45:30 smithi183 ceph-mon[25285]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:45:30.705 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:45:30 smithi183 ceph-mon[25285]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:45:30.705 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:45:30 smithi183 ceph-mon[31860]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-03-20T08:45:30.705 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:45:30 smithi183 ceph-mon[31860]: osdmap e52: 8 total, 7 up, 8 in 2024-03-20T08:45:30.705 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:45:30 smithi183 ceph-mon[31860]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-20T08:45:30.705 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:45:30 smithi183 ceph-mon[31860]: from='osd.7 [v2:172.21.15.194:6824/3170101984,v1:172.21.15.194:6825/3170101984]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi194", "root=default"]}]: dispatch 2024-03-20T08:45:30.705 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:45:30 smithi183 ceph-mon[31860]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi194", "root=default"]}]: dispatch 2024-03-20T08:45:30.705 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:45:30 smithi183 ceph-mon[31860]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:45:30.705 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:45:30 smithi183 ceph-mon[31860]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:45:30.705 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:45:30 smithi183 ceph-mon[31860]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:45:31.194 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-20T08:45:31.389 INFO:journalctl@ceph.osd.7.smithi194.stdout:Mar 20 08:45:30 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-osd-7[44872]: 2024-03-20T08:45:30.954+0000 7f37dcb74700 -1 osd.7 0 waiting for initial osdmap 2024-03-20T08:45:31.389 INFO:journalctl@ceph.osd.7.smithi194.stdout:Mar 20 08:45:30 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-osd-7[44872]: 2024-03-20T08:45:30.960+0000 7f37d415e700 -1 osd.7 53 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-20T08:45:31.832 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:45:31 smithi183 ceph-mon[25285]: Detected new or changed devices on smithi194 2024-03-20T08:45:31.832 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:45:31 smithi183 ceph-mon[25285]: pgmap v187: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail 2024-03-20T08:45:31.832 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:45:31 smithi183 ceph-mon[25285]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi194", "root=default"]}]': finished 2024-03-20T08:45:31.832 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:45:31 smithi183 ceph-mon[25285]: osdmap e53: 8 total, 7 up, 8 in 2024-03-20T08:45:31.832 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:45:31 smithi183 ceph-mon[25285]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-20T08:45:31.833 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:45:31 smithi183 ceph-mon[25285]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-20T08:45:31.833 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:45:31 smithi183 ceph-mon[25285]: from='osd.7 ' entity='osd.7' 2024-03-20T08:45:31.833 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:45:31 smithi183 ceph-mon[25285]: from='client.? 172.21.15.183:0/1959646135' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-03-20T08:45:31.833 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:45:31 smithi183 ceph-mon[25285]: osd.7 [v2:172.21.15.194:6824/3170101984,v1:172.21.15.194:6825/3170101984] boot 2024-03-20T08:45:31.833 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:45:31 smithi183 ceph-mon[25285]: osdmap e54: 8 total, 8 up, 8 in 2024-03-20T08:45:31.833 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:45:31 smithi183 ceph-mon[25285]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-20T08:45:31.833 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:45:31 smithi183 ceph-mon[31860]: Detected new or changed devices on smithi194 2024-03-20T08:45:31.833 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:45:31 smithi183 ceph-mon[31860]: pgmap v187: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail 2024-03-20T08:45:31.833 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:45:31 smithi183 ceph-mon[31860]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi194", "root=default"]}]': finished 2024-03-20T08:45:31.833 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:45:31 smithi183 ceph-mon[31860]: osdmap e53: 8 total, 7 up, 8 in 2024-03-20T08:45:31.833 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:45:31 smithi183 ceph-mon[31860]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-20T08:45:31.833 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:45:31 smithi183 ceph-mon[31860]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-20T08:45:31.834 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:45:31 smithi183 ceph-mon[31860]: from='osd.7 ' entity='osd.7' 2024-03-20T08:45:31.834 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:45:31 smithi183 ceph-mon[31860]: from='client.? 172.21.15.183:0/1959646135' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-03-20T08:45:31.834 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:45:31 smithi183 ceph-mon[31860]: osd.7 [v2:172.21.15.194:6824/3170101984,v1:172.21.15.194:6825/3170101984] boot 2024-03-20T08:45:31.834 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:45:31 smithi183 ceph-mon[31860]: osdmap e54: 8 total, 8 up, 8 in 2024-03-20T08:45:31.834 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:45:31 smithi183 ceph-mon[31860]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-20T08:45:31.859 INFO:teuthology.orchestra.run.smithi183.stdout:{"epoch":53,"num_osds":8,"num_up_osds":7,"osd_up_since":1710924307,"num_in_osds":8,"osd_in_since":1710924312,"num_remapped_pgs":0} 2024-03-20T08:45:31.889 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:45:31 smithi194 ceph-mon[27237]: Detected new or changed devices on smithi194 2024-03-20T08:45:31.889 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:45:31 smithi194 ceph-mon[27237]: pgmap v187: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail 2024-03-20T08:45:31.889 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:45:31 smithi194 ceph-mon[27237]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi194", "root=default"]}]': finished 2024-03-20T08:45:31.889 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:45:31 smithi194 ceph-mon[27237]: osdmap e53: 8 total, 7 up, 8 in 2024-03-20T08:45:31.889 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:45:31 smithi194 ceph-mon[27237]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-20T08:45:31.889 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:45:31 smithi194 ceph-mon[27237]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-20T08:45:31.889 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:45:31 smithi194 ceph-mon[27237]: from='osd.7 ' entity='osd.7' 2024-03-20T08:45:31.889 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:45:31 smithi194 ceph-mon[27237]: from='client.? 172.21.15.183:0/1959646135' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-03-20T08:45:31.889 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:45:31 smithi194 ceph-mon[27237]: osd.7 [v2:172.21.15.194:6824/3170101984,v1:172.21.15.194:6825/3170101984] boot 2024-03-20T08:45:31.889 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:45:31 smithi194 ceph-mon[27237]: osdmap e54: 8 total, 8 up, 8 in 2024-03-20T08:45:31.889 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:45:31 smithi194 ceph-mon[27237]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-20T08:45:32.796 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:45:32 smithi194 ceph-mon[27237]: purged_snaps scrub starts 2024-03-20T08:45:32.796 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:45:32 smithi194 ceph-mon[27237]: purged_snaps scrub ok 2024-03-20T08:45:32.860 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4c3f3eba-e695-11ee-95c9-87774f69a715 -- ceph osd stat -f json 2024-03-20T08:45:32.890 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:45:32 smithi183 ceph-mon[25285]: purged_snaps scrub starts 2024-03-20T08:45:32.890 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:45:32 smithi183 ceph-mon[25285]: purged_snaps scrub ok 2024-03-20T08:45:32.891 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:45:32 smithi183 ceph-mon[31860]: purged_snaps scrub starts 2024-03-20T08:45:32.891 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:45:32 smithi183 ceph-mon[31860]: purged_snaps scrub ok 2024-03-20T08:45:33.108 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/4c3f3eba-e695-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T08:45:33.638 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:45:33 smithi194 ceph-mon[27237]: pgmap v190: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-03-20T08:45:33.638 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:45:33 smithi194 ceph-mon[27237]: osdmap e55: 8 total, 8 up, 8 in 2024-03-20T08:45:33.954 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:45:33 smithi183 ceph-mon[25285]: pgmap v190: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-03-20T08:45:33.954 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:45:33 smithi183 ceph-mon[25285]: osdmap e55: 8 total, 8 up, 8 in 2024-03-20T08:45:33.955 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:45:33 smithi183 ceph-mon[31860]: pgmap v190: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-03-20T08:45:33.955 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:45:33 smithi183 ceph-mon[31860]: osdmap e55: 8 total, 8 up, 8 in 2024-03-20T08:45:34.704 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:45:34 smithi183 ceph-mon[25285]: osdmap e56: 8 total, 8 up, 8 in 2024-03-20T08:45:34.705 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:45:34 smithi183 ceph-mon[31860]: osdmap e56: 8 total, 8 up, 8 in 2024-03-20T08:45:34.719 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:45:34 smithi194 ceph-mon[27237]: osdmap e56: 8 total, 8 up, 8 in 2024-03-20T08:45:34.829 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-20T08:45:35.530 INFO:teuthology.orchestra.run.smithi183.stdout:{"epoch":57,"num_osds":8,"num_up_osds":8,"osd_up_since":1710924331,"num_in_osds":8,"osd_in_since":1710924312,"num_remapped_pgs":0} 2024-03-20T08:45:35.531 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 4c3f3eba-e695-11ee-95c9-87774f69a715 -- ceph osd dump --format=json 2024-03-20T08:45:35.819 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/4c3f3eba-e695-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T08:45:35.888 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:45:35 smithi194 ceph-mon[27237]: pgmap v193: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-03-20T08:45:35.889 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:45:35 smithi194 ceph-mon[27237]: osdmap e57: 8 total, 8 up, 8 in 2024-03-20T08:45:35.889 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:45:35 smithi194 ceph-mon[27237]: from='client.? 172.21.15.183:0/2770291328' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-03-20T08:45:35.940 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:45:35 smithi183 ceph-mon[25285]: pgmap v193: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-03-20T08:45:35.940 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:45:35 smithi183 ceph-mon[25285]: osdmap e57: 8 total, 8 up, 8 in 2024-03-20T08:45:35.941 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:45:35 smithi183 ceph-mon[25285]: from='client.? 172.21.15.183:0/2770291328' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-03-20T08:45:35.941 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:45:35 smithi183 ceph-mon[31860]: pgmap v193: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-03-20T08:45:35.941 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:45:35 smithi183 ceph-mon[31860]: osdmap e57: 8 total, 8 up, 8 in 2024-03-20T08:45:35.941 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:45:35 smithi183 ceph-mon[31860]: from='client.? 172.21.15.183:0/2770291328' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-03-20T08:45:36.888 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:45:36 smithi194 ceph-mon[27237]: osdmap e58: 8 total, 8 up, 8 in 2024-03-20T08:45:36.954 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:45:36 smithi183 ceph-mon[25285]: osdmap e58: 8 total, 8 up, 8 in 2024-03-20T08:45:36.954 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:45:36 smithi183 ceph-mon[31860]: osdmap e58: 8 total, 8 up, 8 in 2024-03-20T08:45:37.704 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:45:37 smithi183 ceph-mon[31860]: pgmap v196: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-03-20T08:45:37.704 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:45:37 smithi183 ceph-mon[31860]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:45:37.704 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:45:37 smithi183 ceph-mon[31860]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:45:37.704 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:45:37 smithi183 ceph-mon[31860]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:45:37.705 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:45:37 smithi183 ceph-mon[25285]: pgmap v196: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-03-20T08:45:37.705 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:45:37 smithi183 ceph-mon[25285]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:45:37.705 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:45:37 smithi183 ceph-mon[25285]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:45:37.705 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:45:37 smithi183 ceph-mon[25285]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:45:37.805 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-20T08:45:37.805 INFO:teuthology.orchestra.run.smithi183.stdout:{"epoch":58,"fsid":"4c3f3eba-e695-11ee-95c9-87774f69a715","created":"2024-03-20T08:40:04.332276+0000","modified":"2024-03-20T08:45:35.505591+0000","last_up_change":"2024-03-20T08:45:31.479100+0000","last_in_change":"2024-03-20T08:45:12.076043+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":18,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-20T08:43:21.611845+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":7.8899998664855957,"score_stable":7.8899998664855957,"optimal_score":0.37999999523162842,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"8ed0dd19-c4b0-4768-b28c-957c19afa5b4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6802","nonce":1252352917},{"type":"v1","addr":"172.21.15.183:6803","nonce":1252352917}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6804","nonce":1252352917},{"type":"v1","addr":"172.21.15.183:6805","nonce":1252352917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6808","nonce":1252352917},{"type":"v1","addr":"172.21.15.183:6809","nonce":1252352917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6806","nonce":1252352917},{"type":"v1","addr":"172.21.15.183:6807","nonce":1252352917}]},"public_addr":"172.21.15.183:6803/1252352917","cluster_addr":"172.21.15.183:6805/1252352917","heartbeat_back_addr":"172.21.15.183:6809/1252352917","heartbeat_front_addr":"172.21.15.183:6807/1252352917","state":["exists","up"]},{"osd":1,"uuid":"c881374c-df10-42e3-8a63-3f06638b1d44","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6810","nonce":3442013001},{"type":"v1","addr":"172.21.15.183:6811","nonce":3442013001}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6812","nonce":3442013001},{"type":"v1","addr":"172.21.15.183:6813","nonce":3442013001}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6816","nonce":3442013001},{"type":"v1","addr":"172.21.15.183:6817","nonce":3442013001}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6814","nonce":3442013001},{"type":"v1","addr":"172.21.15.183:6815","nonce":3442013001}]},"public_addr":"172.21.15.183:6811/3442013001","cluster_addr":"172.21.15.183:6813/3442013001","heartbeat_back_addr":"172.21.15.183:6817/3442013001","heartbeat_front_addr":"172.21.15.183:6815/3442013001","state":["exists","up"]},{"osd":2,"uuid":"c1040fce-5e05-4b4f-85b9-da261546233e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6818","nonce":1715612189},{"type":"v1","addr":"172.21.15.183:6819","nonce":1715612189}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6820","nonce":1715612189},{"type":"v1","addr":"172.21.15.183:6821","nonce":1715612189}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6824","nonce":1715612189},{"type":"v1","addr":"172.21.15.183:6825","nonce":1715612189}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6822","nonce":1715612189},{"type":"v1","addr":"172.21.15.183:6823","nonce":1715612189}]},"public_addr":"172.21.15.183:6819/1715612189","cluster_addr":"172.21.15.183:6821/1715612189","heartbeat_back_addr":"172.21.15.183:6825/1715612189","heartbeat_front_addr":"172.21.15.183:6823/1715612189","state":["exists","up"]},{"osd":3,"uuid":"c47d2b57-e03c-48ce-872a-5458833fc3d1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6826","nonce":4169086406},{"type":"v1","addr":"172.21.15.183:6827","nonce":4169086406}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6828","nonce":4169086406},{"type":"v1","addr":"172.21.15.183:6829","nonce":4169086406}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6832","nonce":4169086406},{"type":"v1","addr":"172.21.15.183:6833","nonce":4169086406}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6830","nonce":4169086406},{"type":"v1","addr":"172.21.15.183:6831","nonce":4169086406}]},"public_addr":"172.21.15.183:6827/4169086406","cluster_addr":"172.21.15.183:6829/4169086406","heartbeat_back_addr":"172.21.15.183:6833/4169086406","heartbeat_front_addr":"172.21.15.183:6831/4169086406","state":["exists","up"]},{"osd":4,"uuid":"38be26c8-fc1b-4f09-8545-1f58b80a38b7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":30,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6800","nonce":2756833656},{"type":"v1","addr":"172.21.15.194:6801","nonce":2756833656}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6802","nonce":2756833656},{"type":"v1","addr":"172.21.15.194:6803","nonce":2756833656}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6806","nonce":2756833656},{"type":"v1","addr":"172.21.15.194:6807","nonce":2756833656}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6804","nonce":2756833656},{"type":"v1","addr":"172.21.15.194:6805","nonce":2756833656}]},"public_addr":"172.21.15.194:6801/2756833656","cluster_addr":"172.21.15.194:6803/2756833656","heartbeat_back_addr":"172.21.15.194:6807/2756833656","heartbeat_front_addr":"172.21.15.194:6805/2756833656","state":["exists","up"]},{"osd":5,"uuid":"dcafefba-cc63-4c52-93a8-6296eed08381","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6808","nonce":1048900641},{"type":"v1","addr":"172.21.15.194:6809","nonce":1048900641}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6810","nonce":1048900641},{"type":"v1","addr":"172.21.15.194:6811","nonce":1048900641}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6814","nonce":1048900641},{"type":"v1","addr":"172.21.15.194:6815","nonce":1048900641}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6812","nonce":1048900641},{"type":"v1","addr":"172.21.15.194:6813","nonce":1048900641}]},"public_addr":"172.21.15.194:6809/1048900641","cluster_addr":"172.21.15.194:6811/1048900641","heartbeat_back_addr":"172.21.15.194:6815/1048900641","heartbeat_front_addr":"172.21.15.194:6813/1048900641","state":["exists","up"]},{"osd":6,"uuid":"44adcd7a-6dbf-4437-911f-b41c87190193","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":49,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6816","nonce":3187912321},{"type":"v1","addr":"172.21.15.194:6817","nonce":3187912321}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6818","nonce":3187912321},{"type":"v1","addr":"172.21.15.194:6819","nonce":3187912321}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6822","nonce":3187912321},{"type":"v1","addr":"172.21.15.194:6823","nonce":3187912321}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6820","nonce":3187912321},{"type":"v1","addr":"172.21.15.194:6821","nonce":3187912321}]},"public_addr":"172.21.15.194:6817/3187912321","cluster_addr":"172.21.15.194:6819/3187912321","heartbeat_back_addr":"172.21.15.194:6823/3187912321","heartbeat_front_addr":"172.21.15.194:6821/3187912321","state":["exists","up"]},{"osd":7,"uuid":"87e8f364-5518-446c-84d8-7443f76a3bd9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":54,"up_thru":57,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6824","nonce":3170101984},{"type":"v1","addr":"172.21.15.194:6825","nonce":3170101984}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6826","nonce":3170101984},{"type":"v1","addr":"172.21.15.194:6827","nonce":3170101984}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6830","nonce":3170101984},{"type":"v1","addr":"172.21.15.194:6831","nonce":3170101984}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6828","nonce":3170101984},{"type":"v1","addr":"172.21.15.194:6829","nonce":3170101984}]},"public_addr":"172.21.15.194:6825/3170101984","cluster_addr":"172.21.15.194:6827/3170101984","heartbeat_back_addr":"172.21.15.194:6831/3170101984","heartbeat_front_addr":"172.21.15.194:6829/3170101984","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-20T08:42:28.471457+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-20T08:42:53.440908+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-20T08:43:19.214787+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-20T08:43:48.797018+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-20T08:44:13.628103+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-20T08:44:39.185881+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-20T08:45:05.248710+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-20T08:45:29.544075+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.183:0/1027467964":"2024-03-21T08:40:59.502103+0000","172.21.15.183:6801/1361970260":"2024-03-21T08:40:59.502103+0000","172.21.15.183:6800/1361970260":"2024-03-21T08:40:59.502103+0000","172.21.15.183:0/3285061906":"2024-03-21T08:40:33.183142+0000","172.21.15.183:6800/2997288523":"2024-03-21T08:40:33.183142+0000","172.21.15.183:0/3209928172":"2024-03-21T08:40:59.502103+0000","172.21.15.183:0/213307139":"2024-03-21T08:40:33.183142+0000","172.21.15.183:6801/2997288523":"2024-03-21T08:40:33.183142+0000","172.21.15.183:0/3451815539":"2024-03-21T08:40:33.183142+0000","172.21.15.183:0/2737646670":"2024-03-21T08:40:59.502103+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-20T08:45:37.888 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:45:37 smithi194 ceph-mon[27237]: pgmap v196: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-03-20T08:45:37.889 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:45:37 smithi194 ceph-mon[27237]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:45:37.889 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:45:37 smithi194 ceph-mon[27237]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:45:37.889 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:45:37 smithi194 ceph-mon[27237]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:45:38.488 INFO:tasks.cephadm.ceph_manager.ceph:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-20T08:43:21.611845+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 7.889999866485596, 'score_stable': 7.889999866485596, 'optimal_score': 0.3799999952316284, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-20T08:45:38.488 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 4c3f3eba-e695-11ee-95c9-87774f69a715 -- ceph osd pool get .mgr pg_num 2024-03-20T08:45:38.784 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:45:38 smithi183 ceph-mon[25285]: from='client.? 172.21.15.183:0/724163973' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-20T08:45:38.784 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:45:38 smithi183 ceph-mon[31860]: from='client.? 172.21.15.183:0/724163973' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-20T08:45:38.888 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:45:38 smithi194 ceph-mon[27237]: from='client.? 172.21.15.183:0/724163973' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-20T08:45:39.366 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/4c3f3eba-e695-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T08:45:39.832 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:45:39 smithi183 ceph-mon[31860]: pgmap v197: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail; 100 KiB/s, 0 objects/s recovering 2024-03-20T08:45:39.833 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:45:39 smithi183 ceph-mon[25285]: pgmap v197: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail; 100 KiB/s, 0 objects/s recovering 2024-03-20T08:45:39.888 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:45:39 smithi194 ceph-mon[27237]: pgmap v197: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail; 100 KiB/s, 0 objects/s recovering 2024-03-20T08:45:41.678 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:45:41 smithi183 ceph-mon[25285]: pgmap v198: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail; 85 KiB/s, 0 objects/s recovering 2024-03-20T08:45:41.678 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:45:41 smithi183 ceph-mon[31860]: pgmap v198: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail; 85 KiB/s, 0 objects/s recovering 2024-03-20T08:45:41.888 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:45:41 smithi194 ceph-mon[27237]: pgmap v198: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail; 85 KiB/s, 0 objects/s recovering 2024-03-20T08:45:42.810 INFO:teuthology.orchestra.run.smithi183.stdout:pg_num: 1 2024-03-20T08:45:43.496 INFO:tasks.cephadm:Adding ceph.rgw.foo.a on smithi183 2024-03-20T08:45:43.496 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4c3f3eba-e695-11ee-95c9-87774f69a715 -- ceph orch apply rgw foo.a --placement '1;smithi183=foo.a' 2024-03-20T08:45:43.752 INFO:teuthology.orchestra.run.smithi194.stderr:Inferring config /var/lib/ceph/4c3f3eba-e695-11ee-95c9-87774f69a715/mon.b/config 2024-03-20T08:45:43.873 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:45:43 smithi194 ceph-mon[27237]: pgmap v199: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-03-20T08:45:43.873 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:45:43 smithi194 ceph-mon[27237]: from='client.? 172.21.15.183:0/205607779' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-20T08:45:43.954 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:45:43 smithi183 ceph-mon[31860]: pgmap v199: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-03-20T08:45:43.954 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:45:43 smithi183 ceph-mon[31860]: from='client.? 172.21.15.183:0/205607779' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-20T08:45:43.954 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:45:43 smithi183 ceph-mon[25285]: pgmap v199: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-03-20T08:45:43.955 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:45:43 smithi183 ceph-mon[25285]: from='client.? 172.21.15.183:0/205607779' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-20T08:45:45.300 INFO:teuthology.orchestra.run.smithi194.stdout:Scheduled rgw.foo.a update... 2024-03-20T08:45:45.855 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:45:45 smithi194 ceph-mon[27237]: pgmap v200: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail; 59 KiB/s, 0 objects/s recovering 2024-03-20T08:45:45.855 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:45:45 smithi194 ceph-mon[27237]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:45:45.855 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:45:45 smithi194 ceph-mon[27237]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-20T08:45:45.856 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:45:45 smithi194 ceph-mon[27237]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-20T08:45:45.856 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:45:45 smithi194 ceph-mon[27237]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-20T08:45:45.856 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:45:45 smithi194 ceph-mon[27237]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:45:45.856 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:45:45 smithi194 ceph-mon[27237]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.a", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-03-20T08:45:45.856 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:45:45 smithi194 ceph-mon[27237]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.a", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-03-20T08:45:45.856 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:45:45 smithi194 ceph-mon[27237]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:45:45.856 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:45:45 smithi194 ceph-mon[27237]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-20T08:45:45.857 DEBUG:teuthology.orchestra.run.smithi183:rgw.foo.a> sudo journalctl -f -n 0 -u ceph-4c3f3eba-e695-11ee-95c9-87774f69a715@rgw.foo.a.service 2024-03-20T08:45:45.859 INFO:tasks.cephadm:Adding ceph.iscsi.iscsi.a on smithi194 2024-03-20T08:45:45.860 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4c3f3eba-e695-11ee-95c9-87774f69a715 -- ceph osd pool create datapool 3 3 replicated 2024-03-20T08:45:45.883 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:45:45 smithi183 ceph-mon[25285]: pgmap v200: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail; 59 KiB/s, 0 objects/s recovering 2024-03-20T08:45:45.883 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:45:45 smithi183 ceph-mon[25285]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:45:45.883 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:45:45 smithi183 ceph-mon[25285]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-20T08:45:45.883 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:45:45 smithi183 ceph-mon[25285]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-20T08:45:45.883 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:45:45 smithi183 ceph-mon[25285]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-20T08:45:45.883 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:45:45 smithi183 ceph-mon[25285]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:45:45.883 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:45:45 smithi183 ceph-mon[25285]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.a", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-03-20T08:45:45.883 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:45:45 smithi183 ceph-mon[25285]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.a", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-03-20T08:45:45.883 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:45:45 smithi183 ceph-mon[25285]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:45:45.884 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:45:45 smithi183 ceph-mon[25285]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-20T08:45:45.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:45:45 smithi183 ceph-mon[31860]: pgmap v200: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail; 59 KiB/s, 0 objects/s recovering 2024-03-20T08:45:45.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:45:45 smithi183 ceph-mon[31860]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:45:45.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:45:45 smithi183 ceph-mon[31860]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-20T08:45:45.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:45:45 smithi183 ceph-mon[31860]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-20T08:45:45.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:45:45 smithi183 ceph-mon[31860]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-20T08:45:45.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:45:45 smithi183 ceph-mon[31860]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:45:45.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:45:45 smithi183 ceph-mon[31860]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.a", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-03-20T08:45:45.884 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:45:45 smithi183 ceph-mon[31860]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.a", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-03-20T08:45:45.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:45:45 smithi183 ceph-mon[31860]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:45:45.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:45:45 smithi183 ceph-mon[31860]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-20T08:45:46.098 INFO:teuthology.orchestra.run.smithi194.stderr:Inferring config /var/lib/ceph/4c3f3eba-e695-11ee-95c9-87774f69a715/mon.b/config 2024-03-20T08:45:46.888 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:45:46 smithi194 ceph-mon[27237]: from='client.24316 -' entity='client.admin' cmd=[{"prefix": "orch apply rgw", "svc_id": "foo.a", "placement": "1;smithi183=foo.a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-20T08:45:46.889 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:45:46 smithi194 ceph-mon[27237]: Saving service rgw.foo.a spec with placement smithi183=foo.a;count:1 2024-03-20T08:45:46.889 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:45:46 smithi194 ceph-mon[27237]: Deploying daemon rgw.foo.a on smithi183 2024-03-20T08:45:46.954 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:45:46 smithi183 ceph-mon[25285]: from='client.24316 -' entity='client.admin' cmd=[{"prefix": "orch apply rgw", "svc_id": "foo.a", "placement": "1;smithi183=foo.a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-20T08:45:46.954 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:45:46 smithi183 ceph-mon[25285]: Saving service rgw.foo.a spec with placement smithi183=foo.a;count:1 2024-03-20T08:45:46.955 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:45:46 smithi183 ceph-mon[25285]: Deploying daemon rgw.foo.a on smithi183 2024-03-20T08:45:46.955 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:45:46 smithi183 ceph-mon[31860]: from='client.24316 -' entity='client.admin' cmd=[{"prefix": "orch apply rgw", "svc_id": "foo.a", "placement": "1;smithi183=foo.a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-20T08:45:46.955 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:45:46 smithi183 ceph-mon[31860]: Saving service rgw.foo.a spec with placement smithi183=foo.a;count:1 2024-03-20T08:45:46.955 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:45:46 smithi183 ceph-mon[31860]: Deploying daemon rgw.foo.a on smithi183 2024-03-20T08:45:47.638 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:45:47 smithi194 ceph-mon[27237]: pgmap v201: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail; 54 KiB/s, 0 objects/s recovering 2024-03-20T08:45:47.871 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:45:47 smithi183 ceph-mon[25285]: pgmap v201: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail; 54 KiB/s, 0 objects/s recovering 2024-03-20T08:45:47.871 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:45:47 smithi183 ceph-mon[31860]: pgmap v201: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail; 54 KiB/s, 0 objects/s recovering 2024-03-20T08:45:47.871 INFO:journalctl@ceph.rgw.foo.a.smithi183.stdout:Mar 20 08:45:47 smithi183 podman[50958]: 2024-03-20 08:45:47.754948656 +0000 UTC m=+0.017631942 image pull 9ceba81fb2c8d8682cb159527017d591637b153b3aa4a390ffa8586db6dcda00 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:976a8b2d06e3c6bfd12488297e8584dd704cc06c3e976c7e0f718c13e238e912 2024-03-20T08:45:47.871 INFO:journalctl@ceph.rgw.foo.a.smithi183.stdout:Mar 20 08:45:47 smithi183 podman[50958]: 2024-03-20 08:45:47.871041291 +0000 UTC m=+0.133724560 container create b6a7f433ee926a5b9cb822a30c765ef8755debd41b6cf58e517d619bdb0b2964 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:976a8b2d06e3c6bfd12488297e8584dd704cc06c3e976c7e0f718c13e238e912, name=ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-rgw-foo-a, CEPH_POINT_RELEASE=, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.build-date=20240307, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, RELEASE=wip-yuri5-testing-2024-03-18-1144-e7e80a6) 2024-03-20T08:45:48.148 INFO:journalctl@ceph.rgw.foo.a.smithi183.stdout:Mar 20 08:45:48 smithi183 podman[50958]: 2024-03-20 08:45:48.090115042 +0000 UTC m=+0.352798325 container init b6a7f433ee926a5b9cb822a30c765ef8755debd41b6cf58e517d619bdb0b2964 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:976a8b2d06e3c6bfd12488297e8584dd704cc06c3e976c7e0f718c13e238e912, name=ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-rgw-foo-a, RELEASE=wip-yuri5-testing-2024-03-18-1144-e7e80a6, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240307, ceph=True, org.label-schema.license=GPLv2, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS) 2024-03-20T08:45:48.148 INFO:journalctl@ceph.rgw.foo.a.smithi183.stdout:Mar 20 08:45:48 smithi183 podman[50958]: 2024-03-20 08:45:48.092793361 +0000 UTC m=+0.355476631 container start b6a7f433ee926a5b9cb822a30c765ef8755debd41b6cf58e517d619bdb0b2964 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:976a8b2d06e3c6bfd12488297e8584dd704cc06c3e976c7e0f718c13e238e912, name=ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-rgw-foo-a, RELEASE=wip-yuri5-testing-2024-03-18-1144-e7e80a6, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240307, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-20T08:45:48.454 INFO:journalctl@ceph.rgw.foo.a.smithi183.stdout:Mar 20 08:45:48 smithi183 bash[50958]: b6a7f433ee926a5b9cb822a30c765ef8755debd41b6cf58e517d619bdb0b2964 2024-03-20T08:45:48.454 INFO:journalctl@ceph.rgw.foo.a.smithi183.stdout:Mar 20 08:45:48 smithi183 systemd[1]: Started Ceph rgw.foo.a for 4c3f3eba-e695-11ee-95c9-87774f69a715. 2024-03-20T08:45:48.621 INFO:teuthology.orchestra.run.smithi194.stderr:pool 'datapool' created 2024-03-20T08:45:48.883 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:45:48 smithi183 ceph-mon[25285]: from='client.? 172.21.15.194:0/2173431924' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]: dispatch 2024-03-20T08:45:48.884 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:45:48 smithi183 ceph-mon[25285]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]: dispatch 2024-03-20T08:45:48.884 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:45:48 smithi183 ceph-mon[25285]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:45:48.884 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:45:48 smithi183 ceph-mon[25285]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:45:48.884 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:45:48 smithi183 ceph-mon[25285]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:45:48.884 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:45:48 smithi183 ceph-mon[25285]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:45:48.884 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:45:48 smithi183 ceph-mon[25285]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:45:48.884 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:45:48 smithi183 ceph-mon[25285]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-20T08:45:48.884 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:45:48 smithi183 ceph-mon[25285]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-20T08:45:48.884 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:45:48 smithi183 ceph-mon[25285]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-20T08:45:48.884 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:45:48 smithi183 ceph-mon[25285]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:45:48.884 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:45:48 smithi183 ceph-mon[25285]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:45:48.884 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:45:48 smithi183 ceph-mon[25285]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:45:48.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:45:48 smithi183 ceph-mon[31860]: from='client.? 172.21.15.194:0/2173431924' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]: dispatch 2024-03-20T08:45:48.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:45:48 smithi183 ceph-mon[31860]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]: dispatch 2024-03-20T08:45:48.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:45:48 smithi183 ceph-mon[31860]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:45:48.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:45:48 smithi183 ceph-mon[31860]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:45:48.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:45:48 smithi183 ceph-mon[31860]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:45:48.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:45:48 smithi183 ceph-mon[31860]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:45:48.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:45:48 smithi183 ceph-mon[31860]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:45:48.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:45:48 smithi183 ceph-mon[31860]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-20T08:45:48.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:45:48 smithi183 ceph-mon[31860]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-20T08:45:48.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:45:48 smithi183 ceph-mon[31860]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-20T08:45:48.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:45:48 smithi183 ceph-mon[31860]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:45:48.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:45:48 smithi183 ceph-mon[31860]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:45:48.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:45:48 smithi183 ceph-mon[31860]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:45:48.888 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:45:48 smithi194 ceph-mon[27237]: from='client.? 172.21.15.194:0/2173431924' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]: dispatch 2024-03-20T08:45:48.889 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:45:48 smithi194 ceph-mon[27237]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]: dispatch 2024-03-20T08:45:48.889 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:45:48 smithi194 ceph-mon[27237]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:45:48.889 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:45:48 smithi194 ceph-mon[27237]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:45:48.889 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:45:48 smithi194 ceph-mon[27237]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:45:48.889 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:45:48 smithi194 ceph-mon[27237]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:45:48.889 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:45:48 smithi194 ceph-mon[27237]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:45:48.889 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:45:48 smithi194 ceph-mon[27237]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-20T08:45:48.889 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:45:48 smithi194 ceph-mon[27237]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-20T08:45:48.889 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:45:48 smithi194 ceph-mon[27237]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-20T08:45:48.889 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:45:48 smithi194 ceph-mon[27237]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:45:48.889 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:45:48 smithi194 ceph-mon[27237]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:45:48.889 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:45:48 smithi194 ceph-mon[27237]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:45:49.157 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4c3f3eba-e695-11ee-95c9-87774f69a715 -- rbd pool init datapool 2024-03-20T08:45:49.398 INFO:teuthology.orchestra.run.smithi194.stderr:Inferring config /var/lib/ceph/4c3f3eba-e695-11ee-95c9-87774f69a715/mon.b/config 2024-03-20T08:45:49.780 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:45:49 smithi194 ceph-mon[27237]: Saving service rgw.foo.a spec with placement smithi183=foo.a;count:1 2024-03-20T08:45:49.780 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:45:49 smithi194 ceph-mon[27237]: pgmap v202: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail; 48 KiB/s, 0 objects/s recovering 2024-03-20T08:45:49.780 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:45:49 smithi194 ceph-mon[27237]: Checking dashboard <-> RGW credentials 2024-03-20T08:45:49.780 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:45:49 smithi194 ceph-mon[27237]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]': finished 2024-03-20T08:45:49.780 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:45:49 smithi194 ceph-mon[27237]: osdmap e59: 8 total, 8 up, 8 in 2024-03-20T08:45:49.780 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:45:49 smithi194 ceph-mon[27237]: from='client.? 172.21.15.183:0/1229649056' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]: dispatch 2024-03-20T08:45:49.780 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:45:49 smithi194 ceph-mon[27237]: from='client.? 172.21.15.183:0/3780810743' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]: dispatch 2024-03-20T08:45:49.781 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:45:49 smithi194 ceph-mon[27237]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:45:49.914 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:45:49 smithi183 ceph-mon[25285]: Saving service rgw.foo.a spec with placement smithi183=foo.a;count:1 2024-03-20T08:45:49.914 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:45:49 smithi183 ceph-mon[25285]: pgmap v202: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail; 48 KiB/s, 0 objects/s recovering 2024-03-20T08:45:49.914 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:45:49 smithi183 ceph-mon[25285]: Checking dashboard <-> RGW credentials 2024-03-20T08:45:49.914 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:45:49 smithi183 ceph-mon[25285]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]': finished 2024-03-20T08:45:49.914 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:45:49 smithi183 ceph-mon[25285]: osdmap e59: 8 total, 8 up, 8 in 2024-03-20T08:45:49.914 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:45:49 smithi183 ceph-mon[25285]: from='client.? 172.21.15.183:0/1229649056' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]: dispatch 2024-03-20T08:45:49.914 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:45:49 smithi183 ceph-mon[25285]: from='client.? 172.21.15.183:0/3780810743' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]: dispatch 2024-03-20T08:45:49.914 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:45:49 smithi183 ceph-mon[25285]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:45:49.914 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:45:49 smithi183 ceph-mon[31860]: Saving service rgw.foo.a spec with placement smithi183=foo.a;count:1 2024-03-20T08:45:49.915 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:45:49 smithi183 ceph-mon[31860]: pgmap v202: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail; 48 KiB/s, 0 objects/s recovering 2024-03-20T08:45:49.915 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:45:49 smithi183 ceph-mon[31860]: Checking dashboard <-> RGW credentials 2024-03-20T08:45:49.915 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:45:49 smithi183 ceph-mon[31860]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]': finished 2024-03-20T08:45:49.915 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:45:49 smithi183 ceph-mon[31860]: osdmap e59: 8 total, 8 up, 8 in 2024-03-20T08:45:49.915 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:45:49 smithi183 ceph-mon[31860]: from='client.? 172.21.15.183:0/1229649056' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]: dispatch 2024-03-20T08:45:49.915 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:45:49 smithi183 ceph-mon[31860]: from='client.? 172.21.15.183:0/3780810743' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]: dispatch 2024-03-20T08:45:49.915 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:45:49 smithi183 ceph-mon[31860]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:45:50.690 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:45:50 smithi194 ceph-mon[27237]: from='client.? 172.21.15.183:0/1229649056' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]': finished 2024-03-20T08:45:50.691 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:45:50 smithi194 ceph-mon[27237]: from='client.? 172.21.15.183:0/3780810743' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]': finished 2024-03-20T08:45:50.691 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:45:50 smithi194 ceph-mon[27237]: osdmap e60: 8 total, 8 up, 8 in 2024-03-20T08:45:50.839 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:45:50 smithi183 ceph-mon[25285]: from='client.? 172.21.15.183:0/1229649056' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]': finished 2024-03-20T08:45:50.839 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:45:50 smithi183 ceph-mon[25285]: from='client.? 172.21.15.183:0/3780810743' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]': finished 2024-03-20T08:45:50.839 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:45:50 smithi183 ceph-mon[25285]: osdmap e60: 8 total, 8 up, 8 in 2024-03-20T08:45:50.840 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:45:50 smithi183 ceph-mon[31860]: from='client.? 172.21.15.183:0/1229649056' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]': finished 2024-03-20T08:45:50.840 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:45:50 smithi183 ceph-mon[31860]: from='client.? 172.21.15.183:0/3780810743' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]': finished 2024-03-20T08:45:50.840 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:45:50 smithi183 ceph-mon[31860]: osdmap e60: 8 total, 8 up, 8 in 2024-03-20T08:45:51.889 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:45:51 smithi194 ceph-mon[27237]: pgmap v205: 36 pgs: 31 unknown, 5 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-03-20T08:45:51.889 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:45:51 smithi194 ceph-mon[27237]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-20T08:45:51.889 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:45:51 smithi194 ceph-mon[27237]: osdmap e61: 8 total, 8 up, 8 in 2024-03-20T08:45:51.889 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:45:51 smithi194 ceph-mon[27237]: from='client.? 172.21.15.183:0/1766499102' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-03-20T08:45:51.889 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:45:51 smithi194 ceph-mon[27237]: from='client.? 172.21.15.183:0/3750501854' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-03-20T08:45:51.889 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:45:51 smithi194 ceph-mon[27237]: from='client.? ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-03-20T08:45:51.889 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:45:51 smithi194 ceph-mon[27237]: from='client.? 172.21.15.194:0/2531829891' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "datapool","app": "rbd"}]: dispatch 2024-03-20T08:45:51.889 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:45:51 smithi194 ceph-mon[27237]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "datapool","app": "rbd"}]: dispatch 2024-03-20T08:45:51.954 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:45:51 smithi183 ceph-mon[25285]: pgmap v205: 36 pgs: 31 unknown, 5 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-03-20T08:45:51.954 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:45:51 smithi183 ceph-mon[25285]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-20T08:45:51.955 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:45:51 smithi183 ceph-mon[25285]: osdmap e61: 8 total, 8 up, 8 in 2024-03-20T08:45:51.955 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:45:51 smithi183 ceph-mon[25285]: from='client.? 172.21.15.183:0/1766499102' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-03-20T08:45:51.955 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:45:51 smithi183 ceph-mon[25285]: from='client.? 172.21.15.183:0/3750501854' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-03-20T08:45:51.955 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:45:51 smithi183 ceph-mon[25285]: from='client.? ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-03-20T08:45:51.955 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:45:51 smithi183 ceph-mon[25285]: from='client.? 172.21.15.194:0/2531829891' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "datapool","app": "rbd"}]: dispatch 2024-03-20T08:45:51.955 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:45:51 smithi183 ceph-mon[25285]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "datapool","app": "rbd"}]: dispatch 2024-03-20T08:45:51.955 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:45:51 smithi183 ceph-mon[31860]: pgmap v205: 36 pgs: 31 unknown, 5 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-03-20T08:45:51.955 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:45:51 smithi183 ceph-mon[31860]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-20T08:45:51.955 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:45:51 smithi183 ceph-mon[31860]: osdmap e61: 8 total, 8 up, 8 in 2024-03-20T08:45:51.955 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:45:51 smithi183 ceph-mon[31860]: from='client.? 172.21.15.183:0/1766499102' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-03-20T08:45:51.955 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:45:51 smithi183 ceph-mon[31860]: from='client.? 172.21.15.183:0/3750501854' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-03-20T08:45:51.956 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:45:51 smithi183 ceph-mon[31860]: from='client.? ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-03-20T08:45:51.956 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:45:51 smithi183 ceph-mon[31860]: from='client.? 172.21.15.194:0/2531829891' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "datapool","app": "rbd"}]: dispatch 2024-03-20T08:45:51.956 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:45:51 smithi183 ceph-mon[31860]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "datapool","app": "rbd"}]: dispatch 2024-03-20T08:45:52.889 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:45:52 smithi194 ceph-mon[27237]: from='client.? 172.21.15.183:0/3750501854' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]': finished 2024-03-20T08:45:52.889 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:45:52 smithi194 ceph-mon[27237]: from='client.? ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]': finished 2024-03-20T08:45:52.889 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:45:52 smithi194 ceph-mon[27237]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "datapool","app": "rbd"}]': finished 2024-03-20T08:45:52.889 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:45:52 smithi194 ceph-mon[27237]: osdmap e62: 8 total, 8 up, 8 in 2024-03-20T08:45:52.889 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:45:52 smithi194 ceph-mon[27237]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:45:52.955 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:45:52 smithi183 ceph-mon[25285]: from='client.? 172.21.15.183:0/3750501854' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]': finished 2024-03-20T08:45:52.955 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:45:52 smithi183 ceph-mon[25285]: from='client.? ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]': finished 2024-03-20T08:45:52.955 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:45:52 smithi183 ceph-mon[25285]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "datapool","app": "rbd"}]': finished 2024-03-20T08:45:52.955 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:45:52 smithi183 ceph-mon[25285]: osdmap e62: 8 total, 8 up, 8 in 2024-03-20T08:45:52.956 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:45:52 smithi183 ceph-mon[25285]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:45:52.956 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:45:52 smithi183 ceph-mon[31860]: from='client.? 172.21.15.183:0/3750501854' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]': finished 2024-03-20T08:45:52.956 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:45:52 smithi183 ceph-mon[31860]: from='client.? ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]': finished 2024-03-20T08:45:52.956 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:45:52 smithi183 ceph-mon[31860]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "datapool","app": "rbd"}]': finished 2024-03-20T08:45:52.956 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:45:52 smithi183 ceph-mon[31860]: osdmap e62: 8 total, 8 up, 8 in 2024-03-20T08:45:52.956 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:45:52 smithi183 ceph-mon[31860]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:45:53.889 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:45:53 smithi194 ceph-mon[27237]: pgmap v208: 68 pgs: 17 creating+peering, 40 active+clean, 11 unknown; 578 KiB data, 222 MiB used, 715 GiB / 715 GiB avail; 3.5 KiB/s rd, 1.5 KiB/s wr, 5 op/s 2024-03-20T08:45:53.889 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:45:53 smithi194 ceph-mon[27237]: osdmap e63: 8 total, 8 up, 8 in 2024-03-20T08:45:53.889 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:45:53 smithi194 ceph-mon[27237]: from='client.? 172.21.15.183:0/1766499102' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-03-20T08:45:53.889 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:45:53 smithi194 ceph-mon[27237]: from='client.? ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-03-20T08:45:53.889 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:45:53 smithi194 ceph-mon[27237]: from='client.? 172.21.15.183:0/3750501854' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-03-20T08:45:53.889 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:45:53 smithi194 ceph-mon[27237]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:45:53.889 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:45:53 smithi194 ceph-mon[27237]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:45:53.889 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:45:53 smithi194 ceph-mon[27237]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:45:53.889 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:45:53 smithi194 ceph-mon[27237]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-03-20T08:45:53.889 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:45:53 smithi194 ceph-mon[27237]: Cluster is now healthy 2024-03-20T08:45:53.954 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:45:53 smithi183 ceph-mon[25285]: pgmap v208: 68 pgs: 17 creating+peering, 40 active+clean, 11 unknown; 578 KiB data, 222 MiB used, 715 GiB / 715 GiB avail; 3.5 KiB/s rd, 1.5 KiB/s wr, 5 op/s 2024-03-20T08:45:53.954 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:45:53 smithi183 ceph-mon[25285]: osdmap e63: 8 total, 8 up, 8 in 2024-03-20T08:45:53.954 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:45:53 smithi183 ceph-mon[25285]: from='client.? 172.21.15.183:0/1766499102' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-03-20T08:45:53.954 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:45:53 smithi183 ceph-mon[25285]: from='client.? ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-03-20T08:45:53.955 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:45:53 smithi183 ceph-mon[25285]: from='client.? 172.21.15.183:0/3750501854' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-03-20T08:45:53.955 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:45:53 smithi183 ceph-mon[25285]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:45:53.955 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:45:53 smithi183 ceph-mon[25285]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:45:53.955 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:45:53 smithi183 ceph-mon[25285]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:45:53.955 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:45:53 smithi183 ceph-mon[25285]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-03-20T08:45:53.955 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:45:53 smithi183 ceph-mon[25285]: Cluster is now healthy 2024-03-20T08:45:53.955 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:45:53 smithi183 ceph-mon[31860]: pgmap v208: 68 pgs: 17 creating+peering, 40 active+clean, 11 unknown; 578 KiB data, 222 MiB used, 715 GiB / 715 GiB avail; 3.5 KiB/s rd, 1.5 KiB/s wr, 5 op/s 2024-03-20T08:45:53.955 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:45:53 smithi183 ceph-mon[31860]: osdmap e63: 8 total, 8 up, 8 in 2024-03-20T08:45:53.955 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:45:53 smithi183 ceph-mon[31860]: from='client.? 172.21.15.183:0/1766499102' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-03-20T08:45:53.955 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:45:53 smithi183 ceph-mon[31860]: from='client.? ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-03-20T08:45:53.955 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:45:53 smithi183 ceph-mon[31860]: from='client.? 172.21.15.183:0/3750501854' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-03-20T08:45:53.955 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:45:53 smithi183 ceph-mon[31860]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:45:53.956 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:45:53 smithi183 ceph-mon[31860]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:45:53.956 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:45:53 smithi183 ceph-mon[31860]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:45:53.956 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:45:53 smithi183 ceph-mon[31860]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-03-20T08:45:53.956 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:45:53 smithi183 ceph-mon[31860]: Cluster is now healthy 2024-03-20T08:45:54.418 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4c3f3eba-e695-11ee-95c9-87774f69a715 -- ceph orch apply iscsi datapool admin admin --trusted_ip_list 172.21.15.194 --placement '1;smithi194=iscsi.a' 2024-03-20T08:45:54.664 INFO:teuthology.orchestra.run.smithi194.stderr:Inferring config /var/lib/ceph/4c3f3eba-e695-11ee-95c9-87774f69a715/mon.b/config 2024-03-20T08:45:54.954 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:45:54 smithi183 ceph-mon[31860]: from='client.? ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]': finished 2024-03-20T08:45:54.955 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:45:54 smithi183 ceph-mon[31860]: from='client.? 172.21.15.183:0/3750501854' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]': finished 2024-03-20T08:45:54.955 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:45:54 smithi183 ceph-mon[31860]: osdmap e64: 8 total, 8 up, 8 in 2024-03-20T08:45:54.955 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:45:54 smithi183 ceph-mon[25285]: from='client.? ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]': finished 2024-03-20T08:45:54.955 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:45:54 smithi183 ceph-mon[25285]: from='client.? 172.21.15.183:0/3750501854' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]': finished 2024-03-20T08:45:54.955 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:45:54 smithi183 ceph-mon[25285]: osdmap e64: 8 total, 8 up, 8 in 2024-03-20T08:45:54.980 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:45:54 smithi194 ceph-mon[27237]: from='client.? ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]': finished 2024-03-20T08:45:54.980 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:45:54 smithi194 ceph-mon[27237]: from='client.? 172.21.15.183:0/3750501854' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]': finished 2024-03-20T08:45:54.980 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:45:54 smithi194 ceph-mon[27237]: osdmap e64: 8 total, 8 up, 8 in 2024-03-20T08:45:55.955 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:45:55 smithi183 ceph-mon[31860]: pgmap v211: 100 pgs: 17 creating+peering, 40 active+clean, 43 unknown; 578 KiB data, 222 MiB used, 715 GiB / 715 GiB avail; 3.5 KiB/s rd, 1.5 KiB/s wr, 5 op/s 2024-03-20T08:45:55.955 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:45:55 smithi183 ceph-mon[31860]: osdmap e65: 8 total, 8 up, 8 in 2024-03-20T08:45:55.955 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:45:55 smithi183 ceph-mon[31860]: from='client.? 172.21.15.183:0/3750501854' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-03-20T08:45:55.955 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:45:55 smithi183 ceph-mon[31860]: from='client.? 172.21.15.183:0/1766499102' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-03-20T08:45:55.955 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:45:55 smithi183 ceph-mon[31860]: from='client.? ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-03-20T08:45:55.955 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:45:55 smithi183 ceph-mon[25285]: pgmap v211: 100 pgs: 17 creating+peering, 40 active+clean, 43 unknown; 578 KiB data, 222 MiB used, 715 GiB / 715 GiB avail; 3.5 KiB/s rd, 1.5 KiB/s wr, 5 op/s 2024-03-20T08:45:55.955 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:45:55 smithi183 ceph-mon[25285]: osdmap e65: 8 total, 8 up, 8 in 2024-03-20T08:45:55.955 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:45:55 smithi183 ceph-mon[25285]: from='client.? 172.21.15.183:0/3750501854' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-03-20T08:45:55.955 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:45:55 smithi183 ceph-mon[25285]: from='client.? 172.21.15.183:0/1766499102' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-03-20T08:45:55.956 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:45:55 smithi183 ceph-mon[25285]: from='client.? ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-03-20T08:45:56.006 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:45:55 smithi194 ceph-mon[27237]: pgmap v211: 100 pgs: 17 creating+peering, 40 active+clean, 43 unknown; 578 KiB data, 222 MiB used, 715 GiB / 715 GiB avail; 3.5 KiB/s rd, 1.5 KiB/s wr, 5 op/s 2024-03-20T08:45:56.006 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:45:55 smithi194 ceph-mon[27237]: osdmap e65: 8 total, 8 up, 8 in 2024-03-20T08:45:56.006 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:45:55 smithi194 ceph-mon[27237]: from='client.? 172.21.15.183:0/3750501854' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-03-20T08:45:56.006 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:45:55 smithi194 ceph-mon[27237]: from='client.? 172.21.15.183:0/1766499102' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-03-20T08:45:56.007 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:45:55 smithi194 ceph-mon[27237]: from='client.? ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-03-20T08:45:56.477 INFO:teuthology.orchestra.run.smithi194.stdout:Scheduled iscsi.datapool update... 2024-03-20T08:45:56.954 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:45:56 smithi183 ceph-mon[25285]: from='client.? 172.21.15.183:0/3750501854' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]': finished 2024-03-20T08:45:56.954 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:45:56 smithi183 ceph-mon[25285]: from='client.? ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]': finished 2024-03-20T08:45:56.955 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:45:56 smithi183 ceph-mon[25285]: osdmap e66: 8 total, 8 up, 8 in 2024-03-20T08:45:56.955 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:45:56 smithi183 ceph-mon[25285]: from='client.? 172.21.15.183:0/1766499102' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-03-20T08:45:56.955 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:45:56 smithi183 ceph-mon[25285]: from='client.? ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-03-20T08:45:56.955 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:45:56 smithi183 ceph-mon[25285]: from='client.? 172.21.15.183:0/3750501854' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-03-20T08:45:56.955 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:45:56 smithi183 ceph-mon[25285]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:45:56.955 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:45:56 smithi183 ceph-mon[31860]: from='client.? 172.21.15.183:0/3750501854' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]': finished 2024-03-20T08:45:56.955 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:45:56 smithi183 ceph-mon[31860]: from='client.? ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]': finished 2024-03-20T08:45:56.955 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:45:56 smithi183 ceph-mon[31860]: osdmap e66: 8 total, 8 up, 8 in 2024-03-20T08:45:56.955 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:45:56 smithi183 ceph-mon[31860]: from='client.? 172.21.15.183:0/1766499102' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-03-20T08:45:56.955 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:45:56 smithi183 ceph-mon[31860]: from='client.? ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-03-20T08:45:56.955 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:45:56 smithi183 ceph-mon[31860]: from='client.? 172.21.15.183:0/3750501854' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-03-20T08:45:56.956 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:45:56 smithi183 ceph-mon[31860]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:45:57.083 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:45:56 smithi194 ceph-mon[27237]: from='client.? 172.21.15.183:0/3750501854' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]': finished 2024-03-20T08:45:57.083 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:45:56 smithi194 ceph-mon[27237]: from='client.? ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]': finished 2024-03-20T08:45:57.083 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:45:56 smithi194 ceph-mon[27237]: osdmap e66: 8 total, 8 up, 8 in 2024-03-20T08:45:57.083 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:45:56 smithi194 ceph-mon[27237]: from='client.? 172.21.15.183:0/1766499102' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-03-20T08:45:57.083 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:45:56 smithi194 ceph-mon[27237]: from='client.? ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-03-20T08:45:57.083 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:45:56 smithi194 ceph-mon[27237]: from='client.? 172.21.15.183:0/3750501854' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-03-20T08:45:57.084 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:45:56 smithi194 ceph-mon[27237]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:45:57.111 INFO:tasks.cephadm:Distributing iscsi-gateway.cfg... 2024-03-20T08:45:57.111 DEBUG:teuthology.orchestra.run.smithi183:> set -ex 2024-03-20T08:45:57.111 DEBUG:teuthology.orchestra.run.smithi183:> sudo dd of=/etc/ceph/iscsi-gateway.cfg 2024-03-20T08:45:57.143 DEBUG:teuthology.orchestra.run.smithi194:> set -ex 2024-03-20T08:45:57.143 DEBUG:teuthology.orchestra.run.smithi194:> sudo dd of=/etc/ceph/iscsi-gateway.cfg 2024-03-20T08:45:57.172 DEBUG:teuthology.orchestra.run.smithi194:iscsi.iscsi.a> sudo journalctl -f -n 0 -u ceph-4c3f3eba-e695-11ee-95c9-87774f69a715@iscsi.iscsi.a.service 2024-03-20T08:45:57.214 INFO:tasks.cephadm:Adding prometheus.a on smithi194 2024-03-20T08:45:57.215 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4c3f3eba-e695-11ee-95c9-87774f69a715 -- ceph orch apply prometheus '1;smithi194=a' 2024-03-20T08:45:57.870 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:45:57 smithi194 ceph-mon[27237]: pgmap v214: 132 pgs: 26 creating+peering, 60 active+clean, 46 unknown; 578 KiB data, 222 MiB used, 715 GiB / 715 GiB avail; 511 B/s wr, 0 op/s 2024-03-20T08:45:57.870 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:45:57 smithi194 ceph-mon[27237]: from='client.24367 -' entity='client.admin' cmd=[{"prefix": "orch apply iscsi", "pool": "datapool", "api_user": "admin", "api_password": "admin", "trusted_ip_list": "172.21.15.194", "placement": "1;smithi194=iscsi.a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-20T08:45:57.870 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:45:57 smithi194 ceph-mon[27237]: Saving service iscsi.datapool spec with placement smithi194=iscsi.a;count:1 2024-03-20T08:45:57.870 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:45:57 smithi194 ceph-mon[27237]: from='client.? ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-03-20T08:45:57.870 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:45:57 smithi194 ceph-mon[27237]: from='client.? 172.21.15.183:0/3750501854' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-03-20T08:45:57.870 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:45:57 smithi194 ceph-mon[27237]: osdmap e67: 8 total, 8 up, 8 in 2024-03-20T08:45:57.870 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:45:57 smithi194 ceph-mon[27237]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:45:57.870 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:45:57 smithi194 ceph-mon[27237]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:45:57.871 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:45:57 smithi194 ceph-mon[27237]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:45:57.871 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:45:57 smithi194 ceph-mon[27237]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.iscsi.a", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-20T08:45:57.871 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:45:57 smithi194 ceph-mon[27237]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.iscsi.iscsi.a", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]': finished 2024-03-20T08:45:57.871 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:45:57 smithi194 ceph-mon[27237]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-20T08:45:57.954 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:45:57 smithi183 ceph-mon[25285]: pgmap v214: 132 pgs: 26 creating+peering, 60 active+clean, 46 unknown; 578 KiB data, 222 MiB used, 715 GiB / 715 GiB avail; 511 B/s wr, 0 op/s 2024-03-20T08:45:57.955 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:45:57 smithi183 ceph-mon[25285]: from='client.24367 -' entity='client.admin' cmd=[{"prefix": "orch apply iscsi", "pool": "datapool", "api_user": "admin", "api_password": "admin", "trusted_ip_list": "172.21.15.194", "placement": "1;smithi194=iscsi.a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-20T08:45:57.955 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:45:57 smithi183 ceph-mon[25285]: Saving service iscsi.datapool spec with placement smithi194=iscsi.a;count:1 2024-03-20T08:45:57.955 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:45:57 smithi183 ceph-mon[25285]: from='client.? ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-03-20T08:45:57.955 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:45:57 smithi183 ceph-mon[25285]: from='client.? 172.21.15.183:0/3750501854' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-03-20T08:45:57.955 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:45:57 smithi183 ceph-mon[25285]: osdmap e67: 8 total, 8 up, 8 in 2024-03-20T08:45:57.955 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:45:57 smithi183 ceph-mon[25285]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:45:57.955 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:45:57 smithi183 ceph-mon[25285]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:45:57.955 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:45:57 smithi183 ceph-mon[25285]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:45:57.955 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:45:57 smithi183 ceph-mon[25285]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.iscsi.a", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-20T08:45:57.955 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:45:57 smithi183 ceph-mon[25285]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.iscsi.iscsi.a", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]': finished 2024-03-20T08:45:57.955 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:45:57 smithi183 ceph-mon[25285]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-20T08:45:57.956 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:45:57 smithi183 ceph-mon[31860]: pgmap v214: 132 pgs: 26 creating+peering, 60 active+clean, 46 unknown; 578 KiB data, 222 MiB used, 715 GiB / 715 GiB avail; 511 B/s wr, 0 op/s 2024-03-20T08:45:57.956 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:45:57 smithi183 ceph-mon[31860]: from='client.24367 -' entity='client.admin' cmd=[{"prefix": "orch apply iscsi", "pool": "datapool", "api_user": "admin", "api_password": "admin", "trusted_ip_list": "172.21.15.194", "placement": "1;smithi194=iscsi.a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-20T08:45:57.956 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:45:57 smithi183 ceph-mon[31860]: Saving service iscsi.datapool spec with placement smithi194=iscsi.a;count:1 2024-03-20T08:45:57.956 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:45:57 smithi183 ceph-mon[31860]: from='client.? ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-03-20T08:45:57.956 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:45:57 smithi183 ceph-mon[31860]: from='client.? 172.21.15.183:0/3750501854' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-03-20T08:45:57.956 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:45:57 smithi183 ceph-mon[31860]: osdmap e67: 8 total, 8 up, 8 in 2024-03-20T08:45:57.956 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:45:57 smithi183 ceph-mon[31860]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:45:57.956 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:45:57 smithi183 ceph-mon[31860]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:45:57.956 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:45:57 smithi183 ceph-mon[31860]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:45:57.956 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:45:57 smithi183 ceph-mon[31860]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.iscsi.a", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-20T08:45:57.956 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:45:57 smithi183 ceph-mon[31860]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.iscsi.iscsi.a", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]': finished 2024-03-20T08:45:57.956 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:45:57 smithi183 ceph-mon[31860]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-20T08:45:58.535 INFO:teuthology.orchestra.run.smithi194.stderr:Inferring config /var/lib/ceph/4c3f3eba-e695-11ee-95c9-87774f69a715/mon.b/config 2024-03-20T08:45:58.888 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:45:58 smithi194 ceph-mon[27237]: Deploying daemon iscsi.iscsi.a on smithi194 2024-03-20T08:45:58.954 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:45:58 smithi183 ceph-mon[25285]: Deploying daemon iscsi.iscsi.a on smithi194 2024-03-20T08:45:58.955 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:45:58 smithi183 ceph-mon[31860]: Deploying daemon iscsi.iscsi.a on smithi194 2024-03-20T08:45:59.954 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:45:59 smithi183 ceph-mon[25285]: pgmap v216: 132 pgs: 9 creating+peering, 123 active+clean; 579 KiB data, 222 MiB used, 715 GiB / 715 GiB avail; 15 KiB/s rd, 1.5 KiB/s wr, 36 op/s 2024-03-20T08:45:59.954 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:45:59 smithi183 ceph-mon[31860]: pgmap v216: 132 pgs: 9 creating+peering, 123 active+clean; 579 KiB data, 222 MiB used, 715 GiB / 715 GiB avail; 15 KiB/s rd, 1.5 KiB/s wr, 36 op/s 2024-03-20T08:46:00.032 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:45:59 smithi194 ceph-mon[27237]: pgmap v216: 132 pgs: 9 creating+peering, 123 active+clean; 579 KiB data, 222 MiB used, 715 GiB / 715 GiB avail; 15 KiB/s rd, 1.5 KiB/s wr, 36 op/s 2024-03-20T08:46:00.540 INFO:teuthology.orchestra.run.smithi194.stdout:Scheduled prometheus update... 2024-03-20T08:46:00.540 INFO:journalctl@ceph.iscsi.iscsi.a.smithi194.stdout:Mar 20 08:46:00 smithi194 podman[47452]: 2024-03-20 08:46:00.443182698 +0000 UTC m=+0.119886819 container create 589372ff43adb43d2f6756641db21093909ce04f47b08e8e7acd6a4f3b5a206f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:976a8b2d06e3c6bfd12488297e8584dd704cc06c3e976c7e0f718c13e238e912, name=ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-iscsi-iscsi-a, GIT_BRANCH=HEAD, ceph=True, org.label-schema.vendor=CentOS, RELEASE=wip-yuri5-testing-2024-03-18-1144-e7e80a6, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307) 2024-03-20T08:46:00.889 INFO:journalctl@ceph.iscsi.iscsi.a.smithi194.stdout:Mar 20 08:46:00 smithi194 podman[47452]: 2024-03-20 08:46:00.628938576 +0000 UTC m=+0.305642700 container init 589372ff43adb43d2f6756641db21093909ce04f47b08e8e7acd6a4f3b5a206f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:976a8b2d06e3c6bfd12488297e8584dd704cc06c3e976c7e0f718c13e238e912, name=ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-iscsi-iscsi-a, RELEASE=wip-yuri5-testing-2024-03-18-1144-e7e80a6, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , ceph=True, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, org.label-schema.build-date=20240307) 2024-03-20T08:46:00.889 INFO:journalctl@ceph.iscsi.iscsi.a.smithi194.stdout:Mar 20 08:46:00 smithi194 podman[47452]: 2024-03-20 08:46:00.632438456 +0000 UTC m=+0.309142581 container start 589372ff43adb43d2f6756641db21093909ce04f47b08e8e7acd6a4f3b5a206f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:976a8b2d06e3c6bfd12488297e8584dd704cc06c3e976c7e0f718c13e238e912, name=ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-iscsi-iscsi-a, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.build-date=20240307, io.buildah.version=1.33.5, RELEASE=wip-yuri5-testing-2024-03-18-1144-e7e80a6, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2) 2024-03-20T08:46:00.889 INFO:journalctl@ceph.iscsi.iscsi.a.smithi194.stdout:Mar 20 08:46:00 smithi194 bash[47452]: 589372ff43adb43d2f6756641db21093909ce04f47b08e8e7acd6a4f3b5a206f 2024-03-20T08:46:00.889 INFO:journalctl@ceph.iscsi.iscsi.a.smithi194.stdout:Mar 20 08:46:00 smithi194 systemd[1]: Started Ceph iscsi.iscsi.a for 4c3f3eba-e695-11ee-95c9-87774f69a715. 2024-03-20T08:46:01.209 INFO:journalctl@ceph.iscsi.iscsi.a.smithi194.stdout:Mar 20 08:46:00 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-iscsi-iscsi-a[47467]: debug Started the configuration object watcher 2024-03-20T08:46:01.209 INFO:journalctl@ceph.iscsi.iscsi.a.smithi194.stdout:Mar 20 08:46:00 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-iscsi-iscsi-a[47467]: debug Checking for config object changes every 1s 2024-03-20T08:46:01.209 INFO:journalctl@ceph.iscsi.iscsi.a.smithi194.stdout:Mar 20 08:46:00 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-iscsi-iscsi-a[47467]: debug Processing osd blocklist entries for this node 2024-03-20T08:46:01.329 DEBUG:teuthology.orchestra.run.smithi194:prometheus.a> sudo journalctl -f -n 0 -u ceph-4c3f3eba-e695-11ee-95c9-87774f69a715@prometheus.a.service 2024-03-20T08:46:01.331 INFO:tasks.cephadm:Adding node-exporter.a on smithi183 2024-03-20T08:46:01.331 INFO:tasks.cephadm:Adding node-exporter.b on smithi194 2024-03-20T08:46:01.331 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4c3f3eba-e695-11ee-95c9-87774f69a715 -- ceph orch apply node-exporter '2;smithi183=a;smithi194=b' 2024-03-20T08:46:01.455 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:46:01 smithi183 ceph-mon[25285]: osdmap e68: 8 total, 8 up, 8 in 2024-03-20T08:46:01.455 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:46:01 smithi183 ceph-mon[25285]: pgmap v218: 132 pgs: 4 creating+peering, 128 active+clean; 579 KiB data, 222 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 2.7 KiB/s wr, 80 op/s 2024-03-20T08:46:01.455 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:46:01 smithi183 ceph-mon[25285]: from='client.24388 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "prometheus", "placement": "1;smithi194=a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-20T08:46:01.455 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:46:01 smithi183 ceph-mon[25285]: Saving service prometheus spec with placement smithi194=a;count:1 2024-03-20T08:46:01.455 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:46:01 smithi183 ceph-mon[25285]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:46:01.455 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:46:01 smithi183 ceph-mon[25285]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:46:01.455 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:46:01 smithi183 ceph-mon[25285]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:46:01.455 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:46:01 smithi183 ceph-mon[25285]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:46:01.455 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:46:01 smithi183 ceph-mon[25285]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:46:01.455 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:46:01 smithi183 ceph-mon[25285]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-20T08:46:01.455 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:46:01 smithi183 ceph-mon[25285]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-20T08:46:01.455 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:46:01 smithi183 ceph-mon[25285]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-20T08:46:01.456 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:46:01 smithi183 ceph-mon[25285]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:46:01.456 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:46:01 smithi183 ceph-mon[25285]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "dashboard iscsi-gateway-list"}]: dispatch 2024-03-20T08:46:01.456 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:46:01 smithi183 ceph-mon[25285]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "dashboard set-iscsi-api-ssl-verification", "value": "true"}]: dispatch 2024-03-20T08:46:01.456 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:46:01 smithi183 ceph-mon[25285]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:46:01.456 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:46:01 smithi183 ceph-mon[25285]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "dashboard iscsi-gateway-add", "name": "smithi194"}]: dispatch 2024-03-20T08:46:01.456 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:46:01 smithi183 ceph-mon[25285]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:46:01.456 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:46:01 smithi183 ceph-mon[25285]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-20T08:46:01.456 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:46:01 smithi183 ceph-mon[25285]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-20T08:46:01.456 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:46:01 smithi183 ceph-mon[25285]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-20T08:46:01.456 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:46:01 smithi183 ceph-mon[31860]: osdmap e68: 8 total, 8 up, 8 in 2024-03-20T08:46:01.456 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:46:01 smithi183 ceph-mon[31860]: pgmap v218: 132 pgs: 4 creating+peering, 128 active+clean; 579 KiB data, 222 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 2.7 KiB/s wr, 80 op/s 2024-03-20T08:46:01.456 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:46:01 smithi183 ceph-mon[31860]: from='client.24388 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "prometheus", "placement": "1;smithi194=a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-20T08:46:01.456 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:46:01 smithi183 ceph-mon[31860]: Saving service prometheus spec with placement smithi194=a;count:1 2024-03-20T08:46:01.457 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:46:01 smithi183 ceph-mon[31860]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:46:01.457 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:46:01 smithi183 ceph-mon[31860]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:46:01.457 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:46:01 smithi183 ceph-mon[31860]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:46:01.457 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:46:01 smithi183 ceph-mon[31860]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:46:01.457 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:46:01 smithi183 ceph-mon[31860]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:46:01.457 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:46:01 smithi183 ceph-mon[31860]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-20T08:46:01.457 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:46:01 smithi183 ceph-mon[31860]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-20T08:46:01.457 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:46:01 smithi183 ceph-mon[31860]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-20T08:46:01.457 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:46:01 smithi183 ceph-mon[31860]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:46:01.457 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:46:01 smithi183 ceph-mon[31860]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "dashboard iscsi-gateway-list"}]: dispatch 2024-03-20T08:46:01.457 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:46:01 smithi183 ceph-mon[31860]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "dashboard set-iscsi-api-ssl-verification", "value": "true"}]: dispatch 2024-03-20T08:46:01.457 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:46:01 smithi183 ceph-mon[31860]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:46:01.457 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:46:01 smithi183 ceph-mon[31860]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "dashboard iscsi-gateway-add", "name": "smithi194"}]: dispatch 2024-03-20T08:46:01.457 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:46:01 smithi183 ceph-mon[31860]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:46:01.458 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:46:01 smithi183 ceph-mon[31860]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-20T08:46:01.458 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:46:01 smithi183 ceph-mon[31860]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-20T08:46:01.458 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:46:01 smithi183 ceph-mon[31860]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-20T08:46:01.544 INFO:journalctl@ceph.iscsi.iscsi.a.smithi194.stdout:Mar 20 08:46:01 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-iscsi-iscsi-a[47467]: debug Reading the configuration object to update local LIO configuration 2024-03-20T08:46:01.544 INFO:journalctl@ceph.iscsi.iscsi.a.smithi194.stdout:Mar 20 08:46:01 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-iscsi-iscsi-a[47467]: debug Configuration does not have an entry for this host(smithi194.front.sepia.ceph.com) - nothing to define to LIO 2024-03-20T08:46:01.544 INFO:journalctl@ceph.iscsi.iscsi.a.smithi194.stdout:Mar 20 08:46:01 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-iscsi-iscsi-a[47467]: debug * Running on http://[::]:5000/ (Press CTRL+C to quit) 2024-03-20T08:46:01.544 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:46:01 smithi194 ceph-mon[27237]: osdmap e68: 8 total, 8 up, 8 in 2024-03-20T08:46:01.545 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:46:01 smithi194 ceph-mon[27237]: pgmap v218: 132 pgs: 4 creating+peering, 128 active+clean; 579 KiB data, 222 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 2.7 KiB/s wr, 80 op/s 2024-03-20T08:46:01.545 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:46:01 smithi194 ceph-mon[27237]: from='client.24388 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "prometheus", "placement": "1;smithi194=a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-20T08:46:01.545 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:46:01 smithi194 ceph-mon[27237]: Saving service prometheus spec with placement smithi194=a;count:1 2024-03-20T08:46:01.545 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:46:01 smithi194 ceph-mon[27237]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:46:01.545 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:46:01 smithi194 ceph-mon[27237]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:46:01.545 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:46:01 smithi194 ceph-mon[27237]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:46:01.545 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:46:01 smithi194 ceph-mon[27237]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:46:01.545 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:46:01 smithi194 ceph-mon[27237]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:46:01.546 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:46:01 smithi194 ceph-mon[27237]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-20T08:46:01.546 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:46:01 smithi194 ceph-mon[27237]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-20T08:46:01.546 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:46:01 smithi194 ceph-mon[27237]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-20T08:46:01.546 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:46:01 smithi194 ceph-mon[27237]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:46:01.546 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:46:01 smithi194 ceph-mon[27237]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "dashboard iscsi-gateway-list"}]: dispatch 2024-03-20T08:46:01.546 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:46:01 smithi194 ceph-mon[27237]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "dashboard set-iscsi-api-ssl-verification", "value": "true"}]: dispatch 2024-03-20T08:46:01.546 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:46:01 smithi194 ceph-mon[27237]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:46:01.546 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:46:01 smithi194 ceph-mon[27237]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "dashboard iscsi-gateway-add", "name": "smithi194"}]: dispatch 2024-03-20T08:46:01.546 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:46:01 smithi194 ceph-mon[27237]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:46:01.546 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:46:01 smithi194 ceph-mon[27237]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-20T08:46:01.546 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:46:01 smithi194 ceph-mon[27237]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-20T08:46:01.546 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:46:01 smithi194 ceph-mon[27237]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-20T08:46:02.481 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:46:02 smithi194 ceph-mon[27237]: Checking pool "datapool" exists for service iscsi.datapool 2024-03-20T08:46:02.481 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:46:02 smithi194 ceph-mon[27237]: Metadata not up to date on all hosts. Skipping non agent specs 2024-03-20T08:46:02.482 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:46:02 smithi194 ceph-mon[27237]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard iscsi-gateway-list"}]: dispatch 2024-03-20T08:46:02.482 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:46:02 smithi194 ceph-mon[27237]: Adding iSCSI gateway http://:@172.21.15.194:5000 to Dashboard 2024-03-20T08:46:02.482 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:46:02 smithi194 ceph-mon[27237]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-iscsi-api-ssl-verification", "value": "true"}]: dispatch 2024-03-20T08:46:02.482 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:46:02 smithi194 ceph-mon[27237]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard iscsi-gateway-add", "name": "smithi194"}]: dispatch 2024-03-20T08:46:02.482 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:46:02 smithi194 ceph-mon[27237]: Checking dashboard <-> RGW credentials 2024-03-20T08:46:02.482 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:46:02 smithi194 ceph-mon[27237]: from='client.? 172.21.15.194:0/4054613680' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-20T08:46:02.482 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:46:02 smithi194 ceph-mon[27237]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:46:02.482 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:46:02 smithi194 ceph-mon[27237]: Metadata not up to date on all hosts. Skipping non agent specs 2024-03-20T08:46:02.704 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:46:02 smithi183 ceph-mon[25285]: Checking pool "datapool" exists for service iscsi.datapool 2024-03-20T08:46:02.705 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:46:02 smithi183 ceph-mon[25285]: Metadata not up to date on all hosts. Skipping non agent specs 2024-03-20T08:46:02.705 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:46:02 smithi183 ceph-mon[25285]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard iscsi-gateway-list"}]: dispatch 2024-03-20T08:46:02.705 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:46:02 smithi183 ceph-mon[25285]: Adding iSCSI gateway http://:@172.21.15.194:5000 to Dashboard 2024-03-20T08:46:02.705 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:46:02 smithi183 ceph-mon[25285]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-iscsi-api-ssl-verification", "value": "true"}]: dispatch 2024-03-20T08:46:02.705 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:46:02 smithi183 ceph-mon[25285]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard iscsi-gateway-add", "name": "smithi194"}]: dispatch 2024-03-20T08:46:02.705 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:46:02 smithi183 ceph-mon[25285]: Checking dashboard <-> RGW credentials 2024-03-20T08:46:02.705 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:46:02 smithi183 ceph-mon[25285]: from='client.? 172.21.15.194:0/4054613680' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-20T08:46:02.705 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:46:02 smithi183 ceph-mon[25285]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:46:02.705 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:46:02 smithi183 ceph-mon[25285]: Metadata not up to date on all hosts. Skipping non agent specs 2024-03-20T08:46:02.705 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:46:02 smithi183 ceph-mon[31860]: Checking pool "datapool" exists for service iscsi.datapool 2024-03-20T08:46:02.705 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:46:02 smithi183 ceph-mon[31860]: Metadata not up to date on all hosts. Skipping non agent specs 2024-03-20T08:46:02.706 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:46:02 smithi183 ceph-mon[31860]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard iscsi-gateway-list"}]: dispatch 2024-03-20T08:46:02.706 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:46:02 smithi183 ceph-mon[31860]: Adding iSCSI gateway http://:@172.21.15.194:5000 to Dashboard 2024-03-20T08:46:02.706 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:46:02 smithi183 ceph-mon[31860]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-iscsi-api-ssl-verification", "value": "true"}]: dispatch 2024-03-20T08:46:02.706 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:46:02 smithi183 ceph-mon[31860]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard iscsi-gateway-add", "name": "smithi194"}]: dispatch 2024-03-20T08:46:02.706 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:46:02 smithi183 ceph-mon[31860]: Checking dashboard <-> RGW credentials 2024-03-20T08:46:02.706 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:46:02 smithi183 ceph-mon[31860]: from='client.? 172.21.15.194:0/4054613680' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-20T08:46:02.706 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:46:02 smithi183 ceph-mon[31860]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:46:02.706 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:46:02 smithi183 ceph-mon[31860]: Metadata not up to date on all hosts. Skipping non agent specs 2024-03-20T08:46:02.783 INFO:teuthology.orchestra.run.smithi194.stderr:Inferring config /var/lib/ceph/4c3f3eba-e695-11ee-95c9-87774f69a715/mon.b/config 2024-03-20T08:46:03.296 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:46:03 smithi194 ceph-mon[27237]: mgrmap e16: y(active, since 5m), standbys: x 2024-03-20T08:46:03.296 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:46:03 smithi194 ceph-mon[27237]: pgmap v219: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 73 KiB/s rd, 5.6 KiB/s wr, 174 op/s 2024-03-20T08:46:03.704 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:46:03 smithi183 ceph-mon[25285]: mgrmap e16: y(active, since 5m), standbys: x 2024-03-20T08:46:03.704 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:46:03 smithi183 ceph-mon[25285]: pgmap v219: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 73 KiB/s rd, 5.6 KiB/s wr, 174 op/s 2024-03-20T08:46:03.705 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:46:03 smithi183 ceph-mon[31860]: mgrmap e16: y(active, since 5m), standbys: x 2024-03-20T08:46:03.705 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:46:03 smithi183 ceph-mon[31860]: pgmap v219: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 73 KiB/s rd, 5.6 KiB/s wr, 174 op/s 2024-03-20T08:46:05.509 INFO:teuthology.orchestra.run.smithi194.stdout:Scheduled node-exporter update... 2024-03-20T08:46:05.851 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:46:05 smithi194 ceph-mon[27237]: pgmap v220: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 60 KiB/s rd, 4.6 KiB/s wr, 143 op/s 2024-03-20T08:46:05.851 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:46:05 smithi194 ceph-mon[27237]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:46:05.851 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:46:05 smithi194 ceph-mon[27237]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:46:05.851 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:46:05 smithi194 ceph-mon[27237]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:46:05.851 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:46:05 smithi194 ceph-mon[27237]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-20T08:46:05.851 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:46:05 smithi194 ceph-mon[27237]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-20T08:46:05.852 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:46:05 smithi194 ceph-mon[27237]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-20T08:46:05.852 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:46:05 smithi194 ceph-mon[27237]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:46:05.954 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:46:05 smithi183 ceph-mon[25285]: pgmap v220: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 60 KiB/s rd, 4.6 KiB/s wr, 143 op/s 2024-03-20T08:46:05.954 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:46:05 smithi183 ceph-mon[25285]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:46:05.955 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:46:05 smithi183 ceph-mon[25285]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:46:05.955 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:46:05 smithi183 ceph-mon[25285]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:46:05.955 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:46:05 smithi183 ceph-mon[25285]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-20T08:46:05.955 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:46:05 smithi183 ceph-mon[25285]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-20T08:46:05.955 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:46:05 smithi183 ceph-mon[25285]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-20T08:46:05.955 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:46:05 smithi183 ceph-mon[25285]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:46:05.955 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:46:05 smithi183 ceph-mon[31860]: pgmap v220: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 60 KiB/s rd, 4.6 KiB/s wr, 143 op/s 2024-03-20T08:46:05.955 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:46:05 smithi183 ceph-mon[31860]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:46:05.955 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:46:05 smithi183 ceph-mon[31860]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:46:05.955 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:46:05 smithi183 ceph-mon[31860]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:46:05.955 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:46:05 smithi183 ceph-mon[31860]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-20T08:46:05.955 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:46:05 smithi183 ceph-mon[31860]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-20T08:46:05.955 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:46:05 smithi183 ceph-mon[31860]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-20T08:46:05.956 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:46:05 smithi183 ceph-mon[31860]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:46:06.465 DEBUG:teuthology.orchestra.run.smithi183:node-exporter.a> sudo journalctl -f -n 0 -u ceph-4c3f3eba-e695-11ee-95c9-87774f69a715@node-exporter.a.service 2024-03-20T08:46:06.468 DEBUG:teuthology.orchestra.run.smithi194:node-exporter.b> sudo journalctl -f -n 0 -u ceph-4c3f3eba-e695-11ee-95c9-87774f69a715@node-exporter.b.service 2024-03-20T08:46:06.470 INFO:tasks.cephadm:Adding alertmanager.a on smithi183 2024-03-20T08:46:06.470 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4c3f3eba-e695-11ee-95c9-87774f69a715 -- ceph orch apply alertmanager '1;smithi183=a' 2024-03-20T08:46:06.857 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:46:06 smithi194 ceph-mon[27237]: from='client.24445 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "node-exporter", "placement": "2;smithi183=a;smithi194=b", "target": ["mon-mgr", ""]}]: dispatch 2024-03-20T08:46:06.857 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:46:06 smithi194 ceph-mon[27237]: Saving service node-exporter spec with placement smithi183=a;smithi194=b;count:2 2024-03-20T08:46:06.857 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:46:06 smithi194 ceph-mon[27237]: Metadata not up to date on all hosts. Skipping non agent specs 2024-03-20T08:46:06.954 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:46:06 smithi183 ceph-mon[25285]: from='client.24445 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "node-exporter", "placement": "2;smithi183=a;smithi194=b", "target": ["mon-mgr", ""]}]: dispatch 2024-03-20T08:46:06.955 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:46:06 smithi183 ceph-mon[25285]: Saving service node-exporter spec with placement smithi183=a;smithi194=b;count:2 2024-03-20T08:46:06.955 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:46:06 smithi183 ceph-mon[25285]: Metadata not up to date on all hosts. Skipping non agent specs 2024-03-20T08:46:06.955 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:46:06 smithi183 ceph-mon[31860]: from='client.24445 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "node-exporter", "placement": "2;smithi183=a;smithi194=b", "target": ["mon-mgr", ""]}]: dispatch 2024-03-20T08:46:06.955 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:46:06 smithi183 ceph-mon[31860]: Saving service node-exporter spec with placement smithi183=a;smithi194=b;count:2 2024-03-20T08:46:06.955 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:46:06 smithi183 ceph-mon[31860]: Metadata not up to date on all hosts. Skipping non agent specs 2024-03-20T08:46:07.304 INFO:teuthology.orchestra.run.smithi194.stderr:Inferring config /var/lib/ceph/4c3f3eba-e695-11ee-95c9-87774f69a715/mon.b/config 2024-03-20T08:46:07.637 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:46:07 smithi194 ceph-mon[27237]: pgmap v221: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 50 KiB/s rd, 3.7 KiB/s wr, 119 op/s 2024-03-20T08:46:07.637 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:46:07 smithi194 ceph-mon[27237]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:46:07.638 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:46:07 smithi194 ceph-mon[27237]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:46:07.638 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:46:07 smithi194 ceph-mon[27237]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:46:07.638 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:46:07 smithi194 ceph-mon[27237]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-20T08:46:07.638 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:46:07 smithi194 ceph-mon[27237]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-20T08:46:07.638 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:46:07 smithi194 ceph-mon[27237]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-20T08:46:07.638 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:46:07 smithi194 ceph-mon[27237]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:46:07.954 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:46:07 smithi183 ceph-mon[25285]: pgmap v221: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 50 KiB/s rd, 3.7 KiB/s wr, 119 op/s 2024-03-20T08:46:07.955 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:46:07 smithi183 ceph-mon[25285]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:46:07.955 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:46:07 smithi183 ceph-mon[25285]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:46:07.955 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:46:07 smithi183 ceph-mon[25285]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:46:07.955 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:46:07 smithi183 ceph-mon[25285]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-20T08:46:07.955 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:46:07 smithi183 ceph-mon[25285]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-20T08:46:07.955 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:46:07 smithi183 ceph-mon[25285]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-20T08:46:07.955 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:46:07 smithi183 ceph-mon[25285]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:46:07.955 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:46:07 smithi183 ceph-mon[31860]: pgmap v221: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 50 KiB/s rd, 3.7 KiB/s wr, 119 op/s 2024-03-20T08:46:07.955 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:46:07 smithi183 ceph-mon[31860]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:46:07.955 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:46:07 smithi183 ceph-mon[31860]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:46:07.955 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:46:07 smithi183 ceph-mon[31860]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:46:07.956 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:46:07 smithi183 ceph-mon[31860]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-20T08:46:07.956 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:46:07 smithi183 ceph-mon[31860]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-20T08:46:07.956 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:46:07 smithi183 ceph-mon[31860]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-20T08:46:07.956 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:46:07 smithi183 ceph-mon[31860]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:46:08.888 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:46:08 smithi194 ceph-mon[27237]: Deploying daemon prometheus.a on smithi194 2024-03-20T08:46:08.954 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:46:08 smithi183 ceph-mon[25285]: Deploying daemon prometheus.a on smithi194 2024-03-20T08:46:08.954 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:46:08 smithi183 ceph-mon[31860]: Deploying daemon prometheus.a on smithi194 2024-03-20T08:46:09.093 INFO:teuthology.orchestra.run.smithi194.stdout:Scheduled alertmanager update... 2024-03-20T08:46:09.697 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:46:09 smithi194 ceph-mon[27237]: pgmap v222: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 42 KiB/s rd, 3.2 KiB/s wr, 98 op/s 2024-03-20T08:46:09.697 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:46:09 smithi194 ceph-mon[27237]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:46:09.697 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:46:09 smithi194 ceph-mon[27237]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:46:09.704 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:46:09 smithi183 ceph-mon[25285]: pgmap v222: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 42 KiB/s rd, 3.2 KiB/s wr, 98 op/s 2024-03-20T08:46:09.704 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:46:09 smithi183 ceph-mon[25285]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:46:09.705 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:46:09 smithi183 ceph-mon[25285]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:46:09.705 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:46:09 smithi183 ceph-mon[31860]: pgmap v222: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 42 KiB/s rd, 3.2 KiB/s wr, 98 op/s 2024-03-20T08:46:09.705 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:46:09 smithi183 ceph-mon[31860]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:46:09.705 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:46:09 smithi183 ceph-mon[31860]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:46:09.724 DEBUG:teuthology.orchestra.run.smithi183:alertmanager.a> sudo journalctl -f -n 0 -u ceph-4c3f3eba-e695-11ee-95c9-87774f69a715@alertmanager.a.service 2024-03-20T08:46:09.727 INFO:tasks.cephadm:Adding grafana.a on smithi194 2024-03-20T08:46:09.728 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4c3f3eba-e695-11ee-95c9-87774f69a715 -- ceph orch apply grafana '1;smithi194=a' 2024-03-20T08:46:09.977 INFO:teuthology.orchestra.run.smithi194.stderr:Inferring config /var/lib/ceph/4c3f3eba-e695-11ee-95c9-87774f69a715/mon.b/config 2024-03-20T08:46:10.888 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:46:10 smithi194 ceph-mon[27237]: from='client.24451 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "alertmanager", "placement": "1;smithi183=a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-20T08:46:10.888 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:46:10 smithi194 ceph-mon[27237]: Saving service alertmanager spec with placement smithi183=a;count:1 2024-03-20T08:46:10.954 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:46:10 smithi183 ceph-mon[25285]: from='client.24451 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "alertmanager", "placement": "1;smithi183=a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-20T08:46:10.954 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:46:10 smithi183 ceph-mon[25285]: Saving service alertmanager spec with placement smithi183=a;count:1 2024-03-20T08:46:10.954 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:46:10 smithi183 ceph-mon[31860]: from='client.24451 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "alertmanager", "placement": "1;smithi183=a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-20T08:46:10.955 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:46:10 smithi183 ceph-mon[31860]: Saving service alertmanager spec with placement smithi183=a;count:1 2024-03-20T08:46:11.184 INFO:journalctl@ceph.iscsi.iscsi.a.smithi194.stdout:Mar 20 08:46:10 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-iscsi-iscsi-a[47467]: debug there is no tcmu-runner data available 2024-03-20T08:46:11.888 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:46:11 smithi194 ceph-mon[27237]: pgmap v223: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 30 KiB/s rd, 2.4 KiB/s wr, 70 op/s 2024-03-20T08:46:11.954 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:46:11 smithi183 ceph-mon[25285]: pgmap v223: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 30 KiB/s rd, 2.4 KiB/s wr, 70 op/s 2024-03-20T08:46:11.954 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:46:11 smithi183 ceph-mon[31860]: pgmap v223: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 30 KiB/s rd, 2.4 KiB/s wr, 70 op/s 2024-03-20T08:46:12.508 INFO:teuthology.orchestra.run.smithi194.stdout:Scheduled grafana update... 2024-03-20T08:46:12.888 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:46:12 smithi194 ceph-mon[27237]: from='client.14559 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-20T08:46:12.889 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:46:12 smithi194 ceph-mon[27237]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:46:12.889 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:46:12 smithi194 ceph-mon[27237]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:46:12.889 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:46:12 smithi194 ceph-mon[27237]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:46:12.889 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:46:12 smithi194 ceph-mon[27237]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:46:12.955 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:46:12 smithi183 ceph-mon[25285]: from='client.14559 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-20T08:46:12.955 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:46:12 smithi183 ceph-mon[25285]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:46:12.955 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:46:12 smithi183 ceph-mon[25285]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:46:12.955 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:46:12 smithi183 ceph-mon[25285]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:46:12.955 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:46:12 smithi183 ceph-mon[25285]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:46:12.955 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:46:12 smithi183 ceph-mon[31860]: from='client.14559 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-20T08:46:12.955 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:46:12 smithi183 ceph-mon[31860]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:46:12.955 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:46:12 smithi183 ceph-mon[31860]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:46:12.955 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:46:12 smithi183 ceph-mon[31860]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:46:12.955 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:46:12 smithi183 ceph-mon[31860]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:46:13.888 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:46:13 smithi194 ceph-mon[27237]: pgmap v224: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 26 KiB/s rd, 2.0 KiB/s wr, 59 op/s 2024-03-20T08:46:13.888 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:46:13 smithi194 ceph-mon[27237]: from='client.24457 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "grafana", "placement": "1;smithi194=a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-20T08:46:13.889 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:46:13 smithi194 ceph-mon[27237]: Saving service grafana spec with placement smithi194=a;count:1 2024-03-20T08:46:13.954 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:46:13 smithi183 ceph-mon[25285]: pgmap v224: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 26 KiB/s rd, 2.0 KiB/s wr, 59 op/s 2024-03-20T08:46:13.954 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:46:13 smithi183 ceph-mon[25285]: from='client.24457 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "grafana", "placement": "1;smithi194=a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-20T08:46:13.954 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:46:13 smithi183 ceph-mon[25285]: Saving service grafana spec with placement smithi194=a;count:1 2024-03-20T08:46:13.954 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:46:13 smithi183 ceph-mon[31860]: pgmap v224: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 26 KiB/s rd, 2.0 KiB/s wr, 59 op/s 2024-03-20T08:46:13.955 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:46:13 smithi183 ceph-mon[31860]: from='client.24457 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "grafana", "placement": "1;smithi194=a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-20T08:46:13.955 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:46:13 smithi183 ceph-mon[31860]: Saving service grafana spec with placement smithi194=a;count:1 2024-03-20T08:46:14.255 DEBUG:teuthology.orchestra.run.smithi194:grafana.a> sudo journalctl -f -n 0 -u ceph-4c3f3eba-e695-11ee-95c9-87774f69a715@grafana.a.service 2024-03-20T08:46:14.257 INFO:tasks.cephadm:Setting up client nodes... 2024-03-20T08:46:14.257 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4c3f3eba-e695-11ee-95c9-87774f69a715 -- ceph auth get-or-create client.0 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2024-03-20T08:46:14.501 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/4c3f3eba-e695-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T08:46:15.879 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:46:15 smithi183 ceph-mon[25285]: pgmap v225: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-20T08:46:15.880 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:46:15 smithi183 ceph-mon[31860]: pgmap v225: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-20T08:46:15.888 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:46:15 smithi194 ceph-mon[27237]: pgmap v225: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-20T08:46:16.404 INFO:teuthology.orchestra.run.smithi183.stdout:[client.0] 2024-03-20T08:46:16.404 INFO:teuthology.orchestra.run.smithi183.stdout: key = AQBYovplA6DuFxAAR/1YySxSHoszJNGodkDhGw== 2024-03-20T08:46:16.678 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:46:16 smithi183 ceph-mon[25285]: from='client.? 172.21.15.183:0/1844080543' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-20T08:46:16.678 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:46:16 smithi183 ceph-mon[25285]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-20T08:46:16.678 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:46:16 smithi183 ceph-mon[31860]: from='client.? 172.21.15.183:0/1844080543' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-20T08:46:16.679 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:46:16 smithi183 ceph-mon[31860]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-20T08:46:16.679 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:46:16 smithi183 ceph-mon[31860]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-03-20T08:46:16.796 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:46:16 smithi194 ceph-mon[27237]: from='client.? 172.21.15.183:0/1844080543' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-20T08:46:16.796 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:46:16 smithi194 ceph-mon[27237]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-20T08:46:16.796 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:46:16 smithi194 ceph-mon[27237]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-03-20T08:46:16.954 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:46:16 smithi183 ceph-mon[25285]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-03-20T08:46:17.204 DEBUG:teuthology.orchestra.run.smithi183:> set -ex 2024-03-20T08:46:17.204 DEBUG:teuthology.orchestra.run.smithi183:> sudo dd of=/etc/ceph/ceph.client.0.keyring 2024-03-20T08:46:17.204 DEBUG:teuthology.orchestra.run.smithi183:> sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-03-20T08:46:17.248 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4c3f3eba-e695-11ee-95c9-87774f69a715 -- ceph auth get-or-create client.1 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2024-03-20T08:46:17.887 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:46:17 smithi194 ceph-mon[27237]: pgmap v226: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-20T08:46:17.954 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:46:17 smithi183 ceph-mon[25285]: pgmap v226: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-20T08:46:17.954 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:46:17 smithi183 ceph-mon[31860]: pgmap v226: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-20T08:46:19.089 INFO:teuthology.orchestra.run.smithi194.stderr:Inferring config /var/lib/ceph/4c3f3eba-e695-11ee-95c9-87774f69a715/mon.b/config 2024-03-20T08:46:19.807 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:46:19 smithi194 ceph-mon[27237]: pgmap v227: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-20T08:46:19.954 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:46:19 smithi183 ceph-mon[25285]: pgmap v227: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-20T08:46:19.954 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:46:19 smithi183 ceph-mon[31860]: pgmap v227: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-20T08:46:21.139 INFO:journalctl@ceph.iscsi.iscsi.a.smithi194.stdout:Mar 20 08:46:20 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-iscsi-iscsi-a[47467]: debug there is no tcmu-runner data available 2024-03-20T08:46:21.221 INFO:teuthology.orchestra.run.smithi194.stdout:[client.1] 2024-03-20T08:46:21.222 INFO:teuthology.orchestra.run.smithi194.stdout: key = AQBdovplqg8CDRAAnpYM/5zITzW09ssdZpfvQg== 2024-03-20T08:46:21.954 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:46:21 smithi183 ceph-mon[25285]: pgmap v228: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-20T08:46:21.955 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:46:21 smithi183 ceph-mon[25285]: from='client.? 172.21.15.194:0/3753015356' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-20T08:46:21.955 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:46:21 smithi183 ceph-mon[25285]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-20T08:46:21.955 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:46:21 smithi183 ceph-mon[25285]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-03-20T08:46:21.955 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:46:21 smithi183 ceph-mon[25285]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:46:21.955 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:46:21 smithi183 ceph-mon[25285]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:46:21.955 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:46:21 smithi183 ceph-mon[25285]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:46:21.955 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:46:21 smithi183 ceph-mon[25285]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "mgr module enable", "module": "prometheus"}]: dispatch 2024-03-20T08:46:21.955 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:46:21 smithi183 ceph-mon[31860]: pgmap v228: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-20T08:46:21.955 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:46:21 smithi183 ceph-mon[31860]: from='client.? 172.21.15.194:0/3753015356' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-20T08:46:21.955 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:46:21 smithi183 ceph-mon[31860]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-20T08:46:21.955 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:46:21 smithi183 ceph-mon[31860]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-03-20T08:46:21.956 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:46:21 smithi183 ceph-mon[31860]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:46:21.956 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:46:21 smithi183 ceph-mon[31860]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:46:21.956 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:46:21 smithi183 ceph-mon[31860]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:46:21.956 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:46:21 smithi183 ceph-mon[31860]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "mgr module enable", "module": "prometheus"}]: dispatch 2024-03-20T08:46:22.139 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:46:21 smithi194 ceph-mon[27237]: pgmap v228: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-20T08:46:22.139 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:46:21 smithi194 ceph-mon[27237]: from='client.? 172.21.15.194:0/3753015356' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-20T08:46:22.139 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:46:21 smithi194 ceph-mon[27237]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-20T08:46:22.139 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:46:21 smithi194 ceph-mon[27237]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-03-20T08:46:22.139 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:46:21 smithi194 ceph-mon[27237]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:46:22.139 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:46:21 smithi194 ceph-mon[27237]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:46:22.139 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:46:21 smithi194 ceph-mon[27237]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' 2024-03-20T08:46:22.139 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:46:21 smithi194 ceph-mon[27237]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd=[{"prefix": "mgr module enable", "module": "prometheus"}]: dispatch 2024-03-20T08:46:22.183 DEBUG:teuthology.orchestra.run.smithi194:> set -ex 2024-03-20T08:46:22.183 DEBUG:teuthology.orchestra.run.smithi194:> sudo dd of=/etc/ceph/ceph.client.1.keyring 2024-03-20T08:46:22.183 DEBUG:teuthology.orchestra.run.smithi194:> sudo chmod 0644 /etc/ceph/ceph.client.1.keyring 2024-03-20T08:46:22.227 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-03-20T08:46:22.228 INFO:tasks.cephadm.ceph_manager.ceph:waiting for mgr available 2024-03-20T08:46:22.228 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 4c3f3eba-e695-11ee-95c9-87774f69a715 -- ceph mgr dump --format=json 2024-03-20T08:46:22.483 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/4c3f3eba-e695-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T08:46:22.607 INFO:journalctl@ceph.mgr.y.smithi183.stdout:Mar 20 08:46:22 smithi183 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-mgr-y[25513]: ignoring --setuser ceph since I am not root 2024-03-20T08:46:22.607 INFO:journalctl@ceph.mgr.y.smithi183.stdout:Mar 20 08:46:22 smithi183 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-mgr-y[25513]: ignoring --setgroup ceph since I am not root 2024-03-20T08:46:22.888 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:46:22 smithi183 ceph-mon[25285]: from='client.14559 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-20T08:46:22.888 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:46:22 smithi183 ceph-mon[25285]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd='[{"prefix": "mgr module enable", "module": "prometheus"}]': finished 2024-03-20T08:46:22.888 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:46:22 smithi183 ceph-mon[25285]: mgrmap e17: y(active, since 5m), standbys: x 2024-03-20T08:46:22.888 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:46:22 smithi183 ceph-mon[31860]: from='client.14559 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-20T08:46:22.889 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:46:22 smithi183 ceph-mon[31860]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd='[{"prefix": "mgr module enable", "module": "prometheus"}]': finished 2024-03-20T08:46:22.889 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:46:22 smithi183 ceph-mon[31860]: mgrmap e17: y(active, since 5m), standbys: x 2024-03-20T08:46:22.889 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:46:22 smithi194 ceph-mon[27237]: from='client.14559 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-20T08:46:22.889 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:46:22 smithi194 ceph-mon[27237]: from='mgr.14152 172.21.15.183:0/1836559101' entity='mgr.y' cmd='[{"prefix": "mgr module enable", "module": "prometheus"}]': finished 2024-03-20T08:46:22.889 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:46:22 smithi194 ceph-mon[27237]: mgrmap e17: y(active, since 5m), standbys: x 2024-03-20T08:46:22.889 INFO:journalctl@ceph.mgr.x.smithi194.stdout:Mar 20 08:46:22 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-mgr-x[29672]: ignoring --setuser ceph since I am not root 2024-03-20T08:46:22.890 INFO:journalctl@ceph.mgr.x.smithi194.stdout:Mar 20 08:46:22 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-mgr-x[29672]: ignoring --setgroup ceph since I am not root 2024-03-20T08:46:22.890 INFO:journalctl@ceph.mgr.x.smithi194.stdout:Mar 20 08:46:22 smithi194 ceph-mgr[29676]: -- 172.21.15.194:0/3387331771 <== mon.1 v2:172.21.15.194:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x556d89e085a0 con 0x556d89ddc800 2024-03-20T08:46:23.204 INFO:journalctl@ceph.mgr.y.smithi183.stdout:Mar 20 08:46:23 smithi183 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-mgr-y[25513]: 2024-03-20T08:46:23.034+0000 7fcb231d5200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-20T08:46:23.204 INFO:journalctl@ceph.mgr.y.smithi183.stdout:Mar 20 08:46:23 smithi183 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-mgr-y[25513]: 2024-03-20T08:46:23.129+0000 7fcb231d5200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-20T08:46:23.234 INFO:journalctl@ceph.mgr.x.smithi194.stdout:Mar 20 08:46:23 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-mgr-x[29672]: 2024-03-20T08:46:23.016+0000 7fbeb394c200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-20T08:46:23.234 INFO:journalctl@ceph.mgr.x.smithi194.stdout:Mar 20 08:46:23 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-mgr-x[29672]: 2024-03-20T08:46:23.115+0000 7fbeb394c200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-20T08:46:23.934 INFO:journalctl@ceph.mgr.y.smithi183.stdout:Mar 20 08:46:23 smithi183 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-mgr-y[25513]: 2024-03-20T08:46:23.784+0000 7fcb231d5200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-20T08:46:23.934 INFO:journalctl@ceph.mgr.y.smithi183.stdout:Mar 20 08:46:23 smithi183 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-mgr-y[25513]: 2024-03-20T08:46:23.889+0000 7fcb231d5200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-20T08:46:24.063 INFO:journalctl@ceph.mgr.x.smithi194.stdout:Mar 20 08:46:23 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-mgr-x[29672]: 2024-03-20T08:46:23.766+0000 7fbeb394c200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-20T08:46:24.063 INFO:journalctl@ceph.mgr.x.smithi194.stdout:Mar 20 08:46:23 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-mgr-x[29672]: 2024-03-20T08:46:23.860+0000 7fbeb394c200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-20T08:46:24.063 INFO:journalctl@ceph.mgr.x.smithi194.stdout:Mar 20 08:46:24 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-mgr-x[29672]: 2024-03-20T08:46:24.011+0000 7fbeb394c200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-20T08:46:24.187 INFO:journalctl@ceph.mgr.y.smithi183.stdout:Mar 20 08:46:24 smithi183 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-mgr-y[25513]: 2024-03-20T08:46:24.035+0000 7fcb231d5200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-20T08:46:24.188 INFO:journalctl@ceph.mgr.y.smithi183.stdout:Mar 20 08:46:24 smithi183 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-mgr-y[25513]: 2024-03-20T08:46:24.134+0000 7fcb231d5200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-20T08:46:24.389 INFO:journalctl@ceph.mgr.x.smithi194.stdout:Mar 20 08:46:24 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-mgr-x[29672]: 2024-03-20T08:46:24.108+0000 7fbeb394c200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-20T08:46:24.389 INFO:journalctl@ceph.mgr.x.smithi194.stdout:Mar 20 08:46:24 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-mgr-x[29672]: 2024-03-20T08:46:24.191+0000 7fbeb394c200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-20T08:46:24.454 INFO:journalctl@ceph.mgr.y.smithi183.stdout:Mar 20 08:46:24 smithi183 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-mgr-y[25513]: 2024-03-20T08:46:24.220+0000 7fcb231d5200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-20T08:46:24.594 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-20T08:46:24.642 INFO:journalctl@ceph.mgr.x.smithi194.stdout:Mar 20 08:46:24 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-mgr-x[29672]: 2024-03-20T08:46:24.463+0000 7fbeb394c200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-20T08:46:24.642 INFO:journalctl@ceph.mgr.x.smithi194.stdout:Mar 20 08:46:24 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-mgr-x[29672]: 2024-03-20T08:46:24.576+0000 7fbeb394c200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-20T08:46:24.824 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:46:24 smithi183 ceph-mon[25285]: from='client.? 172.21.15.183:0/1849982865' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-03-20T08:46:24.824 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:46:24 smithi183 ceph-mon[31860]: from='client.? 172.21.15.183:0/1849982865' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-03-20T08:46:24.825 INFO:journalctl@ceph.mgr.y.smithi183.stdout:Mar 20 08:46:24 smithi183 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-mgr-y[25513]: 2024-03-20T08:46:24.517+0000 7fcb231d5200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-20T08:46:24.825 INFO:journalctl@ceph.mgr.y.smithi183.stdout:Mar 20 08:46:24 smithi183 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-mgr-y[25513]: 2024-03-20T08:46:24.634+0000 7fcb231d5200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-20T08:46:24.825 INFO:journalctl@ceph.mgr.y.smithi183.stdout:Mar 20 08:46:24 smithi183 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-mgr-y[25513]: 2024-03-20T08:46:24.725+0000 7fcb231d5200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-20T08:46:24.957 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:46:24 smithi194 ceph-mon[27237]: from='client.? 172.21.15.183:0/1849982865' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-03-20T08:46:24.958 INFO:journalctl@ceph.mgr.x.smithi194.stdout:Mar 20 08:46:24 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-mgr-x[29672]: 2024-03-20T08:46:24.667+0000 7fbeb394c200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-20T08:46:24.958 INFO:journalctl@ceph.mgr.x.smithi194.stdout:Mar 20 08:46:24 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-mgr-x[29672]: 2024-03-20T08:46:24.769+0000 7fbeb394c200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-20T08:46:25.083 INFO:journalctl@ceph.mgr.y.smithi183.stdout:Mar 20 08:46:24 smithi183 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-mgr-y[25513]: 2024-03-20T08:46:24.823+0000 7fcb231d5200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-20T08:46:25.084 INFO:journalctl@ceph.mgr.y.smithi183.stdout:Mar 20 08:46:25 smithi183 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-mgr-y[25513]: 2024-03-20T08:46:25.012+0000 7fcb231d5200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-20T08:46:25.312 INFO:journalctl@ceph.mgr.x.smithi194.stdout:Mar 20 08:46:24 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-mgr-x[29672]: 2024-03-20T08:46:24.957+0000 7fbeb394c200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-20T08:46:25.404 INFO:teuthology.orchestra.run.smithi183.stdout:{"epoch":17,"flags":0,"active_gid":14152,"active_name":"y","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6800","nonce":3255438358},{"type":"v1","addr":"172.21.15.183:6801","nonce":3255438358}]},"active_addr":"172.21.15.183:6801/3255438358","active_change":"2024-03-20T08:40:59.502214+0000","active_mgr_features":4540701547738038271,"available":true,"standbys":[{"gid":24115,"name":"x","mgr_features":4540701547738038271,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.0.0","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["cephadm","dashboard","iostat","nfs","prometheus","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.0.0","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{"dashboard":"https://172.21.15.183:8443/"},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"squid":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":3,"active_clients":[{"name":"cephadm","addrvec":[{"type":"v2","addr":"172.21.15.183:0","nonce":3241368708}]},{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.183:0","nonce":1999141339}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.183:0","nonce":2903520159}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.183:0","nonce":1814559177}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.183:0","nonce":3555591867}]}]} 2024-03-20T08:46:25.410 INFO:tasks.cephadm.ceph_manager.ceph:mgr available! 2024-03-20T08:46:25.410 INFO:tasks.cephadm.ceph_manager.ceph:waiting for all up 2024-03-20T08:46:25.411 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 4c3f3eba-e695-11ee-95c9-87774f69a715 -- ceph osd dump --format=json 2024-03-20T08:46:25.664 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/4c3f3eba-e695-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T08:46:26.102 INFO:journalctl@ceph.mgr.y.smithi183.stdout:Mar 20 08:46:26 smithi183 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-mgr-y[25513]: 2024-03-20T08:46:26.011+0000 7fcb231d5200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-20T08:46:26.390 INFO:journalctl@ceph.mgr.x.smithi194.stdout:Mar 20 08:46:25 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-mgr-x[29672]: 2024-03-20T08:46:25.958+0000 7fbeb394c200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-20T08:46:26.390 INFO:journalctl@ceph.mgr.x.smithi194.stdout:Mar 20 08:46:26 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-mgr-x[29672]: 2024-03-20T08:46:26.106+0000 7fbeb394c200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-20T08:46:26.454 INFO:journalctl@ceph.mgr.y.smithi183.stdout:Mar 20 08:46:26 smithi183 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-mgr-y[25513]: 2024-03-20T08:46:26.147+0000 7fcb231d5200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-20T08:46:26.772 INFO:journalctl@ceph.mgr.x.smithi194.stdout:Mar 20 08:46:26 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-mgr-x[29672]: 2024-03-20T08:46:26.470+0000 7fbeb394c200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-20T08:46:26.772 INFO:journalctl@ceph.mgr.x.smithi194.stdout:Mar 20 08:46:26 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-mgr-x[29672]: 2024-03-20T08:46:26.566+0000 7fbeb394c200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-20T08:46:26.772 INFO:journalctl@ceph.mgr.x.smithi194.stdout:Mar 20 08:46:26 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-mgr-x[29672]: 2024-03-20T08:46:26.771+0000 7fbeb394c200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-20T08:46:26.776 INFO:journalctl@ceph.mgr.y.smithi183.stdout:Mar 20 08:46:26 smithi183 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-mgr-y[25513]: 2024-03-20T08:46:26.487+0000 7fcb231d5200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-20T08:46:26.776 INFO:journalctl@ceph.mgr.y.smithi183.stdout:Mar 20 08:46:26 smithi183 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-mgr-y[25513]: 2024-03-20T08:46:26.576+0000 7fcb231d5200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-20T08:46:27.123 INFO:journalctl@ceph.mgr.y.smithi183.stdout:Mar 20 08:46:26 smithi183 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-mgr-y[25513]: 2024-03-20T08:46:26.775+0000 7fcb231d5200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-20T08:46:27.123 INFO:journalctl@ceph.mgr.y.smithi183.stdout:Mar 20 08:46:26 smithi183 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-mgr-y[25513]: 2024-03-20T08:46:26.971+0000 7fcb231d5200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-20T08:46:27.139 INFO:journalctl@ceph.mgr.x.smithi194.stdout:Mar 20 08:46:26 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-mgr-x[29672]: 2024-03-20T08:46:26.956+0000 7fbeb394c200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-20T08:46:27.422 INFO:journalctl@ceph.mgr.y.smithi183.stdout:Mar 20 08:46:27 smithi183 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-mgr-y[25513]: 2024-03-20T08:46:27.286+0000 7fcb231d5200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-20T08:46:27.507 INFO:journalctl@ceph.mgr.x.smithi194.stdout:Mar 20 08:46:27 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-mgr-x[29672]: 2024-03-20T08:46:27.239+0000 7fbeb394c200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-20T08:46:27.647 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-20T08:46:27.647 INFO:teuthology.orchestra.run.smithi183.stdout:{"epoch":68,"fsid":"4c3f3eba-e695-11ee-95c9-87774f69a715","created":"2024-03-20T08:40:04.332276+0000","modified":"2024-03-20T08:46:00.203201+0000","last_up_change":"2024-03-20T08:45:31.479100+0000","last_in_change":"2024-03-20T08:45:12.076043+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":18,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":6,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-20T08:43:21.611845+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":7.8899998664855957,"score_stable":7.8899998664855957,"optimal_score":0.37999999523162842,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"datapool","create_time":"2024-03-20T08:45:47.697696+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":3,"pg_placement_num":3,"pg_placement_num_target":3,"pg_num_target":3,"pg_num_pending":3,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"64","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":64,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.6500000953674316,"score_stable":2.6500000953674316,"optimal_score":0.87999999523162842,"raw_score_acting":2.3299999237060547,"raw_score_stable":2.3299999237060547,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":".rgw.root","create_time":"2024-03-20T08:45:48.129891+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"61","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"default.rgw.log","create_time":"2024-03-20T08:45:49.693383+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"63","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"default.rgw.control","create_time":"2024-03-20T08:45:51.656476+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"65","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"default.rgw.meta","create_time":"2024-03-20T08:45:53.721327+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"67","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"8ed0dd19-c4b0-4768-b28c-957c19afa5b4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6802","nonce":1252352917},{"type":"v1","addr":"172.21.15.183:6803","nonce":1252352917}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6804","nonce":1252352917},{"type":"v1","addr":"172.21.15.183:6805","nonce":1252352917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6808","nonce":1252352917},{"type":"v1","addr":"172.21.15.183:6809","nonce":1252352917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6806","nonce":1252352917},{"type":"v1","addr":"172.21.15.183:6807","nonce":1252352917}]},"public_addr":"172.21.15.183:6803/1252352917","cluster_addr":"172.21.15.183:6805/1252352917","heartbeat_back_addr":"172.21.15.183:6809/1252352917","heartbeat_front_addr":"172.21.15.183:6807/1252352917","state":["exists","up"]},{"osd":1,"uuid":"c881374c-df10-42e3-8a63-3f06638b1d44","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6810","nonce":3442013001},{"type":"v1","addr":"172.21.15.183:6811","nonce":3442013001}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6812","nonce":3442013001},{"type":"v1","addr":"172.21.15.183:6813","nonce":3442013001}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6816","nonce":3442013001},{"type":"v1","addr":"172.21.15.183:6817","nonce":3442013001}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6814","nonce":3442013001},{"type":"v1","addr":"172.21.15.183:6815","nonce":3442013001}]},"public_addr":"172.21.15.183:6811/3442013001","cluster_addr":"172.21.15.183:6813/3442013001","heartbeat_back_addr":"172.21.15.183:6817/3442013001","heartbeat_front_addr":"172.21.15.183:6815/3442013001","state":["exists","up"]},{"osd":2,"uuid":"c1040fce-5e05-4b4f-85b9-da261546233e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6818","nonce":1715612189},{"type":"v1","addr":"172.21.15.183:6819","nonce":1715612189}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6820","nonce":1715612189},{"type":"v1","addr":"172.21.15.183:6821","nonce":1715612189}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6824","nonce":1715612189},{"type":"v1","addr":"172.21.15.183:6825","nonce":1715612189}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6822","nonce":1715612189},{"type":"v1","addr":"172.21.15.183:6823","nonce":1715612189}]},"public_addr":"172.21.15.183:6819/1715612189","cluster_addr":"172.21.15.183:6821/1715612189","heartbeat_back_addr":"172.21.15.183:6825/1715612189","heartbeat_front_addr":"172.21.15.183:6823/1715612189","state":["exists","up"]},{"osd":3,"uuid":"c47d2b57-e03c-48ce-872a-5458833fc3d1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6826","nonce":4169086406},{"type":"v1","addr":"172.21.15.183:6827","nonce":4169086406}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6828","nonce":4169086406},{"type":"v1","addr":"172.21.15.183:6829","nonce":4169086406}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6832","nonce":4169086406},{"type":"v1","addr":"172.21.15.183:6833","nonce":4169086406}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6830","nonce":4169086406},{"type":"v1","addr":"172.21.15.183:6831","nonce":4169086406}]},"public_addr":"172.21.15.183:6827/4169086406","cluster_addr":"172.21.15.183:6829/4169086406","heartbeat_back_addr":"172.21.15.183:6833/4169086406","heartbeat_front_addr":"172.21.15.183:6831/4169086406","state":["exists","up"]},{"osd":4,"uuid":"38be26c8-fc1b-4f09-8545-1f58b80a38b7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":30,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6800","nonce":2756833656},{"type":"v1","addr":"172.21.15.194:6801","nonce":2756833656}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6802","nonce":2756833656},{"type":"v1","addr":"172.21.15.194:6803","nonce":2756833656}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6806","nonce":2756833656},{"type":"v1","addr":"172.21.15.194:6807","nonce":2756833656}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6804","nonce":2756833656},{"type":"v1","addr":"172.21.15.194:6805","nonce":2756833656}]},"public_addr":"172.21.15.194:6801/2756833656","cluster_addr":"172.21.15.194:6803/2756833656","heartbeat_back_addr":"172.21.15.194:6807/2756833656","heartbeat_front_addr":"172.21.15.194:6805/2756833656","state":["exists","up"]},{"osd":5,"uuid":"dcafefba-cc63-4c52-93a8-6296eed08381","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6808","nonce":1048900641},{"type":"v1","addr":"172.21.15.194:6809","nonce":1048900641}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6810","nonce":1048900641},{"type":"v1","addr":"172.21.15.194:6811","nonce":1048900641}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6814","nonce":1048900641},{"type":"v1","addr":"172.21.15.194:6815","nonce":1048900641}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6812","nonce":1048900641},{"type":"v1","addr":"172.21.15.194:6813","nonce":1048900641}]},"public_addr":"172.21.15.194:6809/1048900641","cluster_addr":"172.21.15.194:6811/1048900641","heartbeat_back_addr":"172.21.15.194:6815/1048900641","heartbeat_front_addr":"172.21.15.194:6813/1048900641","state":["exists","up"]},{"osd":6,"uuid":"44adcd7a-6dbf-4437-911f-b41c87190193","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":63,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6816","nonce":3187912321},{"type":"v1","addr":"172.21.15.194:6817","nonce":3187912321}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6818","nonce":3187912321},{"type":"v1","addr":"172.21.15.194:6819","nonce":3187912321}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6822","nonce":3187912321},{"type":"v1","addr":"172.21.15.194:6823","nonce":3187912321}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6820","nonce":3187912321},{"type":"v1","addr":"172.21.15.194:6821","nonce":3187912321}]},"public_addr":"172.21.15.194:6817/3187912321","cluster_addr":"172.21.15.194:6819/3187912321","heartbeat_back_addr":"172.21.15.194:6823/3187912321","heartbeat_front_addr":"172.21.15.194:6821/3187912321","state":["exists","up"]},{"osd":7,"uuid":"87e8f364-5518-446c-84d8-7443f76a3bd9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":54,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6824","nonce":3170101984},{"type":"v1","addr":"172.21.15.194:6825","nonce":3170101984}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6826","nonce":3170101984},{"type":"v1","addr":"172.21.15.194:6827","nonce":3170101984}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6830","nonce":3170101984},{"type":"v1","addr":"172.21.15.194:6831","nonce":3170101984}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6828","nonce":3170101984},{"type":"v1","addr":"172.21.15.194:6829","nonce":3170101984}]},"public_addr":"172.21.15.194:6825/3170101984","cluster_addr":"172.21.15.194:6827/3170101984","heartbeat_back_addr":"172.21.15.194:6831/3170101984","heartbeat_front_addr":"172.21.15.194:6829/3170101984","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-20T08:42:28.471457+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-20T08:42:53.440908+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-20T08:43:19.214787+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-20T08:43:48.797018+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-20T08:44:13.628103+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-20T08:44:39.185881+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-20T08:45:05.248710+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-20T08:45:29.544075+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.183:0/1027467964":"2024-03-21T08:40:59.502103+0000","172.21.15.183:6801/1361970260":"2024-03-21T08:40:59.502103+0000","172.21.15.183:6800/1361970260":"2024-03-21T08:40:59.502103+0000","172.21.15.183:0/3285061906":"2024-03-21T08:40:33.183142+0000","172.21.15.183:6800/2997288523":"2024-03-21T08:40:33.183142+0000","172.21.15.183:0/3209928172":"2024-03-21T08:40:59.502103+0000","172.21.15.183:0/213307139":"2024-03-21T08:40:33.183142+0000","172.21.15.183:6801/2997288523":"2024-03-21T08:40:33.183142+0000","172.21.15.183:0/3451815539":"2024-03-21T08:40:33.183142+0000","172.21.15.183:0/2737646670":"2024-03-21T08:40:59.502103+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-20T08:46:27.698 INFO:journalctl@ceph.mgr.y.smithi183.stdout:Mar 20 08:46:27 smithi183 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-mgr-y[25513]: 2024-03-20T08:46:27.559+0000 7fcb231d5200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-20T08:46:27.888 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:46:27 smithi194 ceph-mon[27237]: from='client.? 172.21.15.183:0/86365286' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-20T08:46:27.889 INFO:journalctl@ceph.mgr.x.smithi194.stdout:Mar 20 08:46:27 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-mgr-x[29672]: 2024-03-20T08:46:27.507+0000 7fbeb394c200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-20T08:46:27.889 INFO:journalctl@ceph.mgr.x.smithi194.stdout:Mar 20 08:46:27 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-mgr-x[29672]: 2024-03-20T08:46:27.748+0000 7fbeb394c200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-20T08:46:27.954 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:46:27 smithi183 ceph-mon[25285]: from='client.? 172.21.15.183:0/86365286' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-20T08:46:27.954 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:46:27 smithi183 ceph-mon[31860]: from='client.? 172.21.15.183:0/86365286' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-20T08:46:27.955 INFO:journalctl@ceph.mgr.y.smithi183.stdout:Mar 20 08:46:27 smithi183 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-mgr-y[25513]: 2024-03-20T08:46:27.808+0000 7fcb231d5200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-20T08:46:28.341 INFO:tasks.cephadm.ceph_manager.ceph:all up! 2024-03-20T08:46:28.341 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 4c3f3eba-e695-11ee-95c9-87774f69a715 -- ceph osd dump --format=json 2024-03-20T08:46:28.597 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/4c3f3eba-e695-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T08:46:28.888 INFO:journalctl@ceph.mgr.x.smithi194.stdout:Mar 20 08:46:28 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-mgr-x[29672]: 2024-03-20T08:46:28.590+0000 7fbeb394c200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-20T08:46:28.949 INFO:journalctl@ceph.mgr.y.smithi183.stdout:Mar 20 08:46:28 smithi183 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-mgr-y[25513]: 2024-03-20T08:46:28.666+0000 7fcb231d5200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-20T08:46:29.517 INFO:journalctl@ceph.mgr.x.smithi194.stdout:Mar 20 08:46:29 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-mgr-x[29672]: 2024-03-20T08:46:29.189+0000 7fbeb394c200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-20T08:46:29.517 INFO:journalctl@ceph.mgr.x.smithi194.stdout:Mar 20 08:46:29 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-mgr-x[29672]: 2024-03-20T08:46:29.292+0000 7fbeb394c200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-20T08:46:29.517 INFO:journalctl@ceph.mgr.x.smithi194.stdout:Mar 20 08:46:29 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-mgr-x[29672]: 2024-03-20T08:46:29.516+0000 7fbeb394c200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-20T08:46:29.561 INFO:journalctl@ceph.mgr.y.smithi183.stdout:Mar 20 08:46:29 smithi183 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-mgr-y[25513]: 2024-03-20T08:46:29.254+0000 7fcb231d5200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-20T08:46:29.561 INFO:journalctl@ceph.mgr.y.smithi183.stdout:Mar 20 08:46:29 smithi183 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-mgr-y[25513]: 2024-03-20T08:46:29.355+0000 7fcb231d5200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-20T08:46:29.824 INFO:journalctl@ceph.mgr.y.smithi183.stdout:Mar 20 08:46:29 smithi183 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-mgr-y[25513]: 2024-03-20T08:46:29.578+0000 7fcb231d5200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-20T08:46:29.824 INFO:journalctl@ceph.mgr.y.smithi183.stdout:Mar 20 08:46:29 smithi183 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-mgr-y[25513]: [20/Mar/2024:08:46:29] ENGINE Bus STARTING 2024-03-20T08:46:29.824 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:46:29 smithi183 ceph-mon[25285]: Standby manager daemon x restarted 2024-03-20T08:46:29.824 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:46:29 smithi183 ceph-mon[25285]: Standby manager daemon x started 2024-03-20T08:46:29.824 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:46:29 smithi183 ceph-mon[25285]: from='mgr.? 172.21.15.194:0/1972872298' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-03-20T08:46:29.824 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:46:29 smithi183 ceph-mon[25285]: from='mgr.? 172.21.15.194:0/1972872298' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-20T08:46:29.824 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:46:29 smithi183 ceph-mon[25285]: from='mgr.? 172.21.15.194:0/1972872298' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-03-20T08:46:29.825 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:46:29 smithi183 ceph-mon[25285]: from='mgr.? 172.21.15.194:0/1972872298' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-20T08:46:29.825 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:46:29 smithi183 ceph-mon[31860]: Standby manager daemon x restarted 2024-03-20T08:46:29.825 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:46:29 smithi183 ceph-mon[31860]: Standby manager daemon x started 2024-03-20T08:46:29.825 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:46:29 smithi183 ceph-mon[31860]: from='mgr.? 172.21.15.194:0/1972872298' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-03-20T08:46:29.825 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:46:29 smithi183 ceph-mon[31860]: from='mgr.? 172.21.15.194:0/1972872298' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-20T08:46:29.825 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:46:29 smithi183 ceph-mon[31860]: from='mgr.? 172.21.15.194:0/1972872298' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-03-20T08:46:29.825 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:46:29 smithi183 ceph-mon[31860]: from='mgr.? 172.21.15.194:0/1972872298' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-20T08:46:29.834 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:46:29 smithi194 ceph-mon[27237]: Standby manager daemon x restarted 2024-03-20T08:46:29.834 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:46:29 smithi194 ceph-mon[27237]: Standby manager daemon x started 2024-03-20T08:46:29.834 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:46:29 smithi194 ceph-mon[27237]: from='mgr.? 172.21.15.194:0/1972872298' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-03-20T08:46:29.834 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:46:29 smithi194 ceph-mon[27237]: from='mgr.? 172.21.15.194:0/1972872298' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-20T08:46:29.834 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:46:29 smithi194 ceph-mon[27237]: from='mgr.? 172.21.15.194:0/1972872298' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-03-20T08:46:29.834 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:46:29 smithi194 ceph-mon[27237]: from='mgr.? 172.21.15.194:0/1972872298' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-20T08:46:29.835 INFO:journalctl@ceph.mgr.x.smithi194.stdout:Mar 20 08:46:29 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-mgr-x[29672]: [20/Mar/2024:08:46:29] ENGINE Bus STARTING 2024-03-20T08:46:29.835 INFO:journalctl@ceph.mgr.x.smithi194.stdout:Mar 20 08:46:29 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-mgr-x[29672]: CherryPy Checker: 2024-03-20T08:46:29.835 INFO:journalctl@ceph.mgr.x.smithi194.stdout:Mar 20 08:46:29 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-mgr-x[29672]: The Application mounted at '' has an empty config. 2024-03-20T08:46:29.835 INFO:journalctl@ceph.mgr.x.smithi194.stdout:Mar 20 08:46:29 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-mgr-x[29672]: 2024-03-20T08:46:29.835 INFO:journalctl@ceph.mgr.x.smithi194.stdout:Mar 20 08:46:29 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-mgr-x[29672]: [20/Mar/2024:08:46:29] ENGINE Serving on http://:::9283 2024-03-20T08:46:29.835 INFO:journalctl@ceph.mgr.x.smithi194.stdout:Mar 20 08:46:29 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-mgr-x[29672]: [20/Mar/2024:08:46:29] ENGINE Bus STARTED 2024-03-20T08:46:30.173 INFO:journalctl@ceph.mgr.y.smithi183.stdout:Mar 20 08:46:29 smithi183 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-mgr-y[25513]: CherryPy Checker: 2024-03-20T08:46:30.173 INFO:journalctl@ceph.mgr.y.smithi183.stdout:Mar 20 08:46:29 smithi183 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-mgr-y[25513]: The Application mounted at '' has an empty config. 2024-03-20T08:46:30.173 INFO:journalctl@ceph.mgr.y.smithi183.stdout:Mar 20 08:46:29 smithi183 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-mgr-y[25513]: 2024-03-20T08:46:30.173 INFO:journalctl@ceph.mgr.y.smithi183.stdout:Mar 20 08:46:29 smithi183 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-mgr-y[25513]: [20/Mar/2024:08:46:29] ENGINE Serving on http://:::9283 2024-03-20T08:46:30.173 INFO:journalctl@ceph.mgr.y.smithi183.stdout:Mar 20 08:46:29 smithi183 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-mgr-y[25513]: [20/Mar/2024:08:46:29] ENGINE Bus STARTED 2024-03-20T08:46:30.851 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:46:30 smithi183 ceph-mon[25285]: mgrmap e18: y(active, since 5m), standbys: x 2024-03-20T08:46:30.851 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:46:30 smithi183 ceph-mon[25285]: Active manager daemon y restarted 2024-03-20T08:46:30.851 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:46:30 smithi183 ceph-mon[25285]: Activating manager daemon y 2024-03-20T08:46:30.852 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:46:30 smithi183 ceph-mon[25285]: osdmap e69: 8 total, 8 up, 8 in 2024-03-20T08:46:30.852 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:46:30 smithi183 ceph-mon[25285]: mgrmap e19: y(active, starting, since 0.020302s), standbys: x 2024-03-20T08:46:30.852 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:46:30 smithi183 ceph-mon[25285]: from='mgr.14637 172.21.15.183:0/885080955' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-20T08:46:30.852 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:46:30 smithi183 ceph-mon[25285]: from='mgr.14637 172.21.15.183:0/885080955' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-20T08:46:30.852 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:46:30 smithi183 ceph-mon[25285]: from='mgr.14637 172.21.15.183:0/885080955' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-20T08:46:30.852 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:46:30 smithi183 ceph-mon[25285]: from='mgr.14637 172.21.15.183:0/885080955' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-20T08:46:30.852 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:46:30 smithi183 ceph-mon[25285]: from='mgr.14637 172.21.15.183:0/885080955' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-20T08:46:30.852 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:46:30 smithi183 ceph-mon[25285]: from='mgr.14637 172.21.15.183:0/885080955' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-20T08:46:30.852 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:46:30 smithi183 ceph-mon[25285]: from='mgr.14637 172.21.15.183:0/885080955' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-20T08:46:30.852 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:46:30 smithi183 ceph-mon[25285]: from='mgr.14637 172.21.15.183:0/885080955' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-20T08:46:30.852 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:46:30 smithi183 ceph-mon[25285]: from='mgr.14637 172.21.15.183:0/885080955' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-20T08:46:30.852 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:46:30 smithi183 ceph-mon[25285]: from='mgr.14637 172.21.15.183:0/885080955' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-20T08:46:30.852 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:46:30 smithi183 ceph-mon[25285]: from='mgr.14637 172.21.15.183:0/885080955' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-20T08:46:30.853 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:46:30 smithi183 ceph-mon[25285]: from='mgr.14637 172.21.15.183:0/885080955' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-20T08:46:30.853 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:46:30 smithi183 ceph-mon[25285]: from='mgr.14637 172.21.15.183:0/885080955' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-20T08:46:30.853 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:46:30 smithi183 ceph-mon[25285]: from='mgr.14637 172.21.15.183:0/885080955' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-20T08:46:30.853 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:46:30 smithi183 ceph-mon[25285]: from='mgr.14637 172.21.15.183:0/885080955' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-20T08:46:30.853 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:46:30 smithi183 ceph-mon[25285]: from='mgr.14637 172.21.15.183:0/885080955' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-20T08:46:30.853 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:46:30 smithi183 ceph-mon[25285]: Manager daemon y is now available 2024-03-20T08:46:30.853 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:46:30 smithi183 ceph-mon[25285]: from='mgr.14637 ' entity='mgr.y' 2024-03-20T08:46:30.853 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:46:30 smithi183 ceph-mon[25285]: from='mgr.14637 172.21.15.183:0/885080955' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-20T08:46:30.853 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:46:30 smithi183 ceph-mon[25285]: from='mgr.14637 172.21.15.183:0/885080955' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-20T08:46:30.853 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:46:30 smithi183 ceph-mon[25285]: from='mgr.14637 172.21.15.183:0/885080955' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi183", "name": "osd_memory_target"}]: dispatch 2024-03-20T08:46:30.853 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:46:30 smithi183 ceph-mon[25285]: from='mgr.14637 172.21.15.183:0/885080955' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi194", "name": "osd_memory_target"}]: dispatch 2024-03-20T08:46:30.853 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:46:30 smithi183 ceph-mon[25285]: from='mgr.14637 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi183", "name": "osd_memory_target"}]: dispatch 2024-03-20T08:46:30.853 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:46:30 smithi183 ceph-mon[25285]: from='mgr.14637 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi194", "name": "osd_memory_target"}]: dispatch 2024-03-20T08:46:30.854 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:46:30 smithi183 ceph-mon[31860]: mgrmap e18: y(active, since 5m), standbys: x 2024-03-20T08:46:30.854 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:46:30 smithi183 ceph-mon[31860]: Active manager daemon y restarted 2024-03-20T08:46:30.854 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:46:30 smithi183 ceph-mon[31860]: Activating manager daemon y 2024-03-20T08:46:30.854 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:46:30 smithi183 ceph-mon[31860]: osdmap e69: 8 total, 8 up, 8 in 2024-03-20T08:46:30.854 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:46:30 smithi183 ceph-mon[31860]: mgrmap e19: y(active, starting, since 0.020302s), standbys: x 2024-03-20T08:46:30.854 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:46:30 smithi183 ceph-mon[31860]: from='mgr.14637 172.21.15.183:0/885080955' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-20T08:46:30.854 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:46:30 smithi183 ceph-mon[31860]: from='mgr.14637 172.21.15.183:0/885080955' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-20T08:46:30.854 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:46:30 smithi183 ceph-mon[31860]: from='mgr.14637 172.21.15.183:0/885080955' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-20T08:46:30.854 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:46:30 smithi183 ceph-mon[31860]: from='mgr.14637 172.21.15.183:0/885080955' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-20T08:46:30.854 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:46:30 smithi183 ceph-mon[31860]: from='mgr.14637 172.21.15.183:0/885080955' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-20T08:46:30.854 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:46:30 smithi183 ceph-mon[31860]: from='mgr.14637 172.21.15.183:0/885080955' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-20T08:46:30.854 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:46:30 smithi183 ceph-mon[31860]: from='mgr.14637 172.21.15.183:0/885080955' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-20T08:46:30.855 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:46:30 smithi183 ceph-mon[31860]: from='mgr.14637 172.21.15.183:0/885080955' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-20T08:46:30.855 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:46:30 smithi183 ceph-mon[31860]: from='mgr.14637 172.21.15.183:0/885080955' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-20T08:46:30.855 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:46:30 smithi183 ceph-mon[31860]: from='mgr.14637 172.21.15.183:0/885080955' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-20T08:46:30.855 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:46:30 smithi183 ceph-mon[31860]: from='mgr.14637 172.21.15.183:0/885080955' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-20T08:46:30.855 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:46:30 smithi183 ceph-mon[31860]: from='mgr.14637 172.21.15.183:0/885080955' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-20T08:46:30.855 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:46:30 smithi183 ceph-mon[31860]: from='mgr.14637 172.21.15.183:0/885080955' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-20T08:46:30.855 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:46:30 smithi183 ceph-mon[31860]: from='mgr.14637 172.21.15.183:0/885080955' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-20T08:46:30.855 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:46:30 smithi183 ceph-mon[31860]: from='mgr.14637 172.21.15.183:0/885080955' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-20T08:46:30.855 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:46:30 smithi183 ceph-mon[31860]: from='mgr.14637 172.21.15.183:0/885080955' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-20T08:46:30.855 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:46:30 smithi183 ceph-mon[31860]: Manager daemon y is now available 2024-03-20T08:46:30.855 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:46:30 smithi183 ceph-mon[31860]: from='mgr.14637 ' entity='mgr.y' 2024-03-20T08:46:30.855 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:46:30 smithi183 ceph-mon[31860]: from='mgr.14637 172.21.15.183:0/885080955' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-20T08:46:30.855 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:46:30 smithi183 ceph-mon[31860]: from='mgr.14637 172.21.15.183:0/885080955' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-20T08:46:30.855 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:46:30 smithi183 ceph-mon[31860]: from='mgr.14637 172.21.15.183:0/885080955' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi183", "name": "osd_memory_target"}]: dispatch 2024-03-20T08:46:30.855 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:46:30 smithi183 ceph-mon[31860]: from='mgr.14637 172.21.15.183:0/885080955' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi194", "name": "osd_memory_target"}]: dispatch 2024-03-20T08:46:30.856 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:46:30 smithi183 ceph-mon[31860]: from='mgr.14637 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi183", "name": "osd_memory_target"}]: dispatch 2024-03-20T08:46:30.856 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:46:30 smithi183 ceph-mon[31860]: from='mgr.14637 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi194", "name": "osd_memory_target"}]: dispatch 2024-03-20T08:46:30.856 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:46:30 smithi183 ceph-mon[25285]: from='mgr.14637 ' entity='mgr.y' cmd='[{"prefix": "config rm", "who": "osd/host:smithi183", "name": "osd_memory_target"}]': finished 2024-03-20T08:46:30.856 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:46:30 smithi183 ceph-mon[25285]: from='mgr.14637 ' entity='mgr.y' cmd='[{"prefix": "config rm", "who": "osd/host:smithi194", "name": "osd_memory_target"}]': finished 2024-03-20T08:46:30.856 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:46:30 smithi183 ceph-mon[25285]: from='mgr.14637 ' entity='mgr.y' 2024-03-20T08:46:30.856 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:46:30 smithi183 ceph-mon[25285]: from='mgr.14637 172.21.15.183:0/885080955' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-20T08:46:30.856 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:46:30 smithi183 ceph-mon[25285]: from='mgr.14637 172.21.15.183:0/885080955' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-20T08:46:30.856 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:46:30 smithi183 ceph-mon[25285]: from='mgr.14637 172.21.15.183:0/885080955' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-20T08:46:30.856 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:46:30 smithi183 ceph-mon[25285]: Updating smithi183:/etc/ceph/ceph.conf 2024-03-20T08:46:30.856 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:46:30 smithi183 ceph-mon[25285]: Updating smithi194:/etc/ceph/ceph.conf 2024-03-20T08:46:30.856 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:46:30 smithi183 ceph-mon[25285]: from='mgr.14637 172.21.15.183:0/885080955' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-20T08:46:30.857 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:46:30 smithi183 ceph-mon[25285]: from='mgr.14637 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-20T08:46:30.857 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:46:30 smithi183 ceph-mon[25285]: from='mgr.14637 172.21.15.183:0/885080955' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-20T08:46:30.857 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:46:30 smithi183 ceph-mon[25285]: from='mgr.14637 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-20T08:46:30.857 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:46:30 smithi183 ceph-mon[25285]: Updating smithi194:/var/lib/ceph/4c3f3eba-e695-11ee-95c9-87774f69a715/config/ceph.conf 2024-03-20T08:46:30.857 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:46:30 smithi183 ceph-mon[25285]: Updating smithi183:/var/lib/ceph/4c3f3eba-e695-11ee-95c9-87774f69a715/config/ceph.conf 2024-03-20T08:46:30.857 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:46:30 smithi183 ceph-mon[25285]: Updating smithi194:/etc/ceph/ceph.client.admin.keyring 2024-03-20T08:46:30.857 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:46:30 smithi183 ceph-mon[25285]: Updating smithi183:/etc/ceph/ceph.client.admin.keyring 2024-03-20T08:46:30.857 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:46:30 smithi183 ceph-mon[31860]: from='mgr.14637 ' entity='mgr.y' cmd='[{"prefix": "config rm", "who": "osd/host:smithi183", "name": "osd_memory_target"}]': finished 2024-03-20T08:46:30.857 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:46:30 smithi183 ceph-mon[31860]: from='mgr.14637 ' entity='mgr.y' cmd='[{"prefix": "config rm", "who": "osd/host:smithi194", "name": "osd_memory_target"}]': finished 2024-03-20T08:46:30.857 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:46:30 smithi183 ceph-mon[31860]: from='mgr.14637 ' entity='mgr.y' 2024-03-20T08:46:30.857 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:46:30 smithi183 ceph-mon[31860]: from='mgr.14637 172.21.15.183:0/885080955' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-20T08:46:30.857 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:46:30 smithi183 ceph-mon[31860]: from='mgr.14637 172.21.15.183:0/885080955' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-20T08:46:30.857 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:46:30 smithi183 ceph-mon[31860]: from='mgr.14637 172.21.15.183:0/885080955' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-20T08:46:30.858 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:46:30 smithi183 ceph-mon[31860]: Updating smithi183:/etc/ceph/ceph.conf 2024-03-20T08:46:30.858 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:46:30 smithi183 ceph-mon[31860]: Updating smithi194:/etc/ceph/ceph.conf 2024-03-20T08:46:30.858 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:46:30 smithi183 ceph-mon[31860]: from='mgr.14637 172.21.15.183:0/885080955' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-20T08:46:30.858 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:46:30 smithi183 ceph-mon[31860]: from='mgr.14637 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-20T08:46:30.858 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:46:30 smithi183 ceph-mon[31860]: from='mgr.14637 172.21.15.183:0/885080955' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-20T08:46:30.858 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:46:30 smithi183 ceph-mon[31860]: from='mgr.14637 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-20T08:46:30.858 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:46:30 smithi183 ceph-mon[31860]: Updating smithi194:/var/lib/ceph/4c3f3eba-e695-11ee-95c9-87774f69a715/config/ceph.conf 2024-03-20T08:46:30.858 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:46:30 smithi183 ceph-mon[31860]: Updating smithi183:/var/lib/ceph/4c3f3eba-e695-11ee-95c9-87774f69a715/config/ceph.conf 2024-03-20T08:46:30.858 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:46:30 smithi183 ceph-mon[31860]: Updating smithi194:/etc/ceph/ceph.client.admin.keyring 2024-03-20T08:46:30.858 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:46:30 smithi183 ceph-mon[31860]: Updating smithi183:/etc/ceph/ceph.client.admin.keyring 2024-03-20T08:46:30.889 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:46:30 smithi194 ceph-mon[27237]: mgrmap e18: y(active, since 5m), standbys: x 2024-03-20T08:46:30.889 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:46:30 smithi194 ceph-mon[27237]: Active manager daemon y restarted 2024-03-20T08:46:30.889 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:46:30 smithi194 ceph-mon[27237]: Activating manager daemon y 2024-03-20T08:46:30.889 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:46:30 smithi194 ceph-mon[27237]: osdmap e69: 8 total, 8 up, 8 in 2024-03-20T08:46:30.889 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:46:30 smithi194 ceph-mon[27237]: mgrmap e19: y(active, starting, since 0.020302s), standbys: x 2024-03-20T08:46:30.889 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:46:30 smithi194 ceph-mon[27237]: from='mgr.14637 172.21.15.183:0/885080955' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-20T08:46:30.889 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:46:30 smithi194 ceph-mon[27237]: from='mgr.14637 172.21.15.183:0/885080955' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-20T08:46:30.889 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:46:30 smithi194 ceph-mon[27237]: from='mgr.14637 172.21.15.183:0/885080955' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-20T08:46:30.889 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:46:30 smithi194 ceph-mon[27237]: from='mgr.14637 172.21.15.183:0/885080955' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-20T08:46:30.889 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:46:30 smithi194 ceph-mon[27237]: from='mgr.14637 172.21.15.183:0/885080955' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-20T08:46:30.890 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:46:30 smithi194 ceph-mon[27237]: from='mgr.14637 172.21.15.183:0/885080955' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-20T08:46:30.890 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:46:30 smithi194 ceph-mon[27237]: from='mgr.14637 172.21.15.183:0/885080955' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-20T08:46:30.890 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:46:30 smithi194 ceph-mon[27237]: from='mgr.14637 172.21.15.183:0/885080955' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-20T08:46:30.890 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:46:30 smithi194 ceph-mon[27237]: from='mgr.14637 172.21.15.183:0/885080955' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-20T08:46:30.890 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:46:30 smithi194 ceph-mon[27237]: from='mgr.14637 172.21.15.183:0/885080955' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-20T08:46:30.890 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:46:30 smithi194 ceph-mon[27237]: from='mgr.14637 172.21.15.183:0/885080955' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-20T08:46:30.890 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:46:30 smithi194 ceph-mon[27237]: from='mgr.14637 172.21.15.183:0/885080955' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-20T08:46:30.890 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:46:30 smithi194 ceph-mon[27237]: from='mgr.14637 172.21.15.183:0/885080955' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-20T08:46:30.890 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:46:30 smithi194 ceph-mon[27237]: from='mgr.14637 172.21.15.183:0/885080955' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-20T08:46:30.890 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:46:30 smithi194 ceph-mon[27237]: from='mgr.14637 172.21.15.183:0/885080955' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-20T08:46:30.890 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:46:30 smithi194 ceph-mon[27237]: from='mgr.14637 172.21.15.183:0/885080955' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-20T08:46:30.890 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:46:30 smithi194 ceph-mon[27237]: Manager daemon y is now available 2024-03-20T08:46:30.891 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:46:30 smithi194 ceph-mon[27237]: from='mgr.14637 ' entity='mgr.y' 2024-03-20T08:46:30.891 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:46:30 smithi194 ceph-mon[27237]: from='mgr.14637 172.21.15.183:0/885080955' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-20T08:46:30.891 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:46:30 smithi194 ceph-mon[27237]: from='mgr.14637 172.21.15.183:0/885080955' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-20T08:46:30.891 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:46:30 smithi194 ceph-mon[27237]: from='mgr.14637 172.21.15.183:0/885080955' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi183", "name": "osd_memory_target"}]: dispatch 2024-03-20T08:46:30.891 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:46:30 smithi194 ceph-mon[27237]: from='mgr.14637 172.21.15.183:0/885080955' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi194", "name": "osd_memory_target"}]: dispatch 2024-03-20T08:46:30.891 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:46:30 smithi194 ceph-mon[27237]: from='mgr.14637 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi183", "name": "osd_memory_target"}]: dispatch 2024-03-20T08:46:30.891 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:46:30 smithi194 ceph-mon[27237]: from='mgr.14637 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi194", "name": "osd_memory_target"}]: dispatch 2024-03-20T08:46:30.891 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:46:30 smithi194 ceph-mon[27237]: from='mgr.14637 ' entity='mgr.y' cmd='[{"prefix": "config rm", "who": "osd/host:smithi183", "name": "osd_memory_target"}]': finished 2024-03-20T08:46:30.891 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:46:30 smithi194 ceph-mon[27237]: from='mgr.14637 ' entity='mgr.y' cmd='[{"prefix": "config rm", "who": "osd/host:smithi194", "name": "osd_memory_target"}]': finished 2024-03-20T08:46:30.891 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:46:30 smithi194 ceph-mon[27237]: from='mgr.14637 ' entity='mgr.y' 2024-03-20T08:46:30.891 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:46:30 smithi194 ceph-mon[27237]: from='mgr.14637 172.21.15.183:0/885080955' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-20T08:46:30.891 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:46:30 smithi194 ceph-mon[27237]: from='mgr.14637 172.21.15.183:0/885080955' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-20T08:46:30.891 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:46:30 smithi194 ceph-mon[27237]: from='mgr.14637 172.21.15.183:0/885080955' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-20T08:46:30.892 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:46:30 smithi194 ceph-mon[27237]: Updating smithi183:/etc/ceph/ceph.conf 2024-03-20T08:46:30.892 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:46:30 smithi194 ceph-mon[27237]: Updating smithi194:/etc/ceph/ceph.conf 2024-03-20T08:46:30.892 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:46:30 smithi194 ceph-mon[27237]: from='mgr.14637 172.21.15.183:0/885080955' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-20T08:46:30.892 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:46:30 smithi194 ceph-mon[27237]: from='mgr.14637 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-20T08:46:30.892 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:46:30 smithi194 ceph-mon[27237]: from='mgr.14637 172.21.15.183:0/885080955' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-20T08:46:30.892 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:46:30 smithi194 ceph-mon[27237]: from='mgr.14637 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-20T08:46:30.892 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:46:30 smithi194 ceph-mon[27237]: Updating smithi194:/var/lib/ceph/4c3f3eba-e695-11ee-95c9-87774f69a715/config/ceph.conf 2024-03-20T08:46:30.892 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:46:30 smithi194 ceph-mon[27237]: Updating smithi183:/var/lib/ceph/4c3f3eba-e695-11ee-95c9-87774f69a715/config/ceph.conf 2024-03-20T08:46:30.892 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:46:30 smithi194 ceph-mon[27237]: Updating smithi194:/etc/ceph/ceph.client.admin.keyring 2024-03-20T08:46:30.892 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:46:30 smithi194 ceph-mon[27237]: Updating smithi183:/etc/ceph/ceph.client.admin.keyring 2024-03-20T08:46:31.091 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-20T08:46:31.091 INFO:teuthology.orchestra.run.smithi183.stdout:{"epoch":69,"fsid":"4c3f3eba-e695-11ee-95c9-87774f69a715","created":"2024-03-20T08:40:04.332276+0000","modified":"2024-03-20T08:46:29.591858+0000","last_up_change":"2024-03-20T08:45:31.479100+0000","last_in_change":"2024-03-20T08:45:12.076043+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":18,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":6,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-20T08:43:21.611845+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":7.8899998664855957,"score_stable":7.8899998664855957,"optimal_score":0.37999999523162842,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"datapool","create_time":"2024-03-20T08:45:47.697696+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":3,"pg_placement_num":3,"pg_placement_num_target":3,"pg_num_target":3,"pg_num_pending":3,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"64","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":64,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.6500000953674316,"score_stable":2.6500000953674316,"optimal_score":0.87999999523162842,"raw_score_acting":2.3299999237060547,"raw_score_stable":2.3299999237060547,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":".rgw.root","create_time":"2024-03-20T08:45:48.129891+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"61","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"default.rgw.log","create_time":"2024-03-20T08:45:49.693383+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"63","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"default.rgw.control","create_time":"2024-03-20T08:45:51.656476+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"65","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"default.rgw.meta","create_time":"2024-03-20T08:45:53.721327+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"67","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"8ed0dd19-c4b0-4768-b28c-957c19afa5b4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6802","nonce":1252352917},{"type":"v1","addr":"172.21.15.183:6803","nonce":1252352917}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6804","nonce":1252352917},{"type":"v1","addr":"172.21.15.183:6805","nonce":1252352917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6808","nonce":1252352917},{"type":"v1","addr":"172.21.15.183:6809","nonce":1252352917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6806","nonce":1252352917},{"type":"v1","addr":"172.21.15.183:6807","nonce":1252352917}]},"public_addr":"172.21.15.183:6803/1252352917","cluster_addr":"172.21.15.183:6805/1252352917","heartbeat_back_addr":"172.21.15.183:6809/1252352917","heartbeat_front_addr":"172.21.15.183:6807/1252352917","state":["exists","up"]},{"osd":1,"uuid":"c881374c-df10-42e3-8a63-3f06638b1d44","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6810","nonce":3442013001},{"type":"v1","addr":"172.21.15.183:6811","nonce":3442013001}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6812","nonce":3442013001},{"type":"v1","addr":"172.21.15.183:6813","nonce":3442013001}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6816","nonce":3442013001},{"type":"v1","addr":"172.21.15.183:6817","nonce":3442013001}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6814","nonce":3442013001},{"type":"v1","addr":"172.21.15.183:6815","nonce":3442013001}]},"public_addr":"172.21.15.183:6811/3442013001","cluster_addr":"172.21.15.183:6813/3442013001","heartbeat_back_addr":"172.21.15.183:6817/3442013001","heartbeat_front_addr":"172.21.15.183:6815/3442013001","state":["exists","up"]},{"osd":2,"uuid":"c1040fce-5e05-4b4f-85b9-da261546233e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6818","nonce":1715612189},{"type":"v1","addr":"172.21.15.183:6819","nonce":1715612189}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6820","nonce":1715612189},{"type":"v1","addr":"172.21.15.183:6821","nonce":1715612189}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6824","nonce":1715612189},{"type":"v1","addr":"172.21.15.183:6825","nonce":1715612189}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6822","nonce":1715612189},{"type":"v1","addr":"172.21.15.183:6823","nonce":1715612189}]},"public_addr":"172.21.15.183:6819/1715612189","cluster_addr":"172.21.15.183:6821/1715612189","heartbeat_back_addr":"172.21.15.183:6825/1715612189","heartbeat_front_addr":"172.21.15.183:6823/1715612189","state":["exists","up"]},{"osd":3,"uuid":"c47d2b57-e03c-48ce-872a-5458833fc3d1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6826","nonce":4169086406},{"type":"v1","addr":"172.21.15.183:6827","nonce":4169086406}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6828","nonce":4169086406},{"type":"v1","addr":"172.21.15.183:6829","nonce":4169086406}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6832","nonce":4169086406},{"type":"v1","addr":"172.21.15.183:6833","nonce":4169086406}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6830","nonce":4169086406},{"type":"v1","addr":"172.21.15.183:6831","nonce":4169086406}]},"public_addr":"172.21.15.183:6827/4169086406","cluster_addr":"172.21.15.183:6829/4169086406","heartbeat_back_addr":"172.21.15.183:6833/4169086406","heartbeat_front_addr":"172.21.15.183:6831/4169086406","state":["exists","up"]},{"osd":4,"uuid":"38be26c8-fc1b-4f09-8545-1f58b80a38b7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":30,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6800","nonce":2756833656},{"type":"v1","addr":"172.21.15.194:6801","nonce":2756833656}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6802","nonce":2756833656},{"type":"v1","addr":"172.21.15.194:6803","nonce":2756833656}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6806","nonce":2756833656},{"type":"v1","addr":"172.21.15.194:6807","nonce":2756833656}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6804","nonce":2756833656},{"type":"v1","addr":"172.21.15.194:6805","nonce":2756833656}]},"public_addr":"172.21.15.194:6801/2756833656","cluster_addr":"172.21.15.194:6803/2756833656","heartbeat_back_addr":"172.21.15.194:6807/2756833656","heartbeat_front_addr":"172.21.15.194:6805/2756833656","state":["exists","up"]},{"osd":5,"uuid":"dcafefba-cc63-4c52-93a8-6296eed08381","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6808","nonce":1048900641},{"type":"v1","addr":"172.21.15.194:6809","nonce":1048900641}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6810","nonce":1048900641},{"type":"v1","addr":"172.21.15.194:6811","nonce":1048900641}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6814","nonce":1048900641},{"type":"v1","addr":"172.21.15.194:6815","nonce":1048900641}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6812","nonce":1048900641},{"type":"v1","addr":"172.21.15.194:6813","nonce":1048900641}]},"public_addr":"172.21.15.194:6809/1048900641","cluster_addr":"172.21.15.194:6811/1048900641","heartbeat_back_addr":"172.21.15.194:6815/1048900641","heartbeat_front_addr":"172.21.15.194:6813/1048900641","state":["exists","up"]},{"osd":6,"uuid":"44adcd7a-6dbf-4437-911f-b41c87190193","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":63,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6816","nonce":3187912321},{"type":"v1","addr":"172.21.15.194:6817","nonce":3187912321}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6818","nonce":3187912321},{"type":"v1","addr":"172.21.15.194:6819","nonce":3187912321}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6822","nonce":3187912321},{"type":"v1","addr":"172.21.15.194:6823","nonce":3187912321}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6820","nonce":3187912321},{"type":"v1","addr":"172.21.15.194:6821","nonce":3187912321}]},"public_addr":"172.21.15.194:6817/3187912321","cluster_addr":"172.21.15.194:6819/3187912321","heartbeat_back_addr":"172.21.15.194:6823/3187912321","heartbeat_front_addr":"172.21.15.194:6821/3187912321","state":["exists","up"]},{"osd":7,"uuid":"87e8f364-5518-446c-84d8-7443f76a3bd9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":54,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6824","nonce":3170101984},{"type":"v1","addr":"172.21.15.194:6825","nonce":3170101984}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6826","nonce":3170101984},{"type":"v1","addr":"172.21.15.194:6827","nonce":3170101984}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6830","nonce":3170101984},{"type":"v1","addr":"172.21.15.194:6831","nonce":3170101984}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6828","nonce":3170101984},{"type":"v1","addr":"172.21.15.194:6829","nonce":3170101984}]},"public_addr":"172.21.15.194:6825/3170101984","cluster_addr":"172.21.15.194:6827/3170101984","heartbeat_back_addr":"172.21.15.194:6831/3170101984","heartbeat_front_addr":"172.21.15.194:6829/3170101984","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-20T08:42:28.471457+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-20T08:42:53.440908+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-20T08:43:19.214787+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-20T08:43:48.797018+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-20T08:44:13.628103+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-20T08:44:39.185881+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-20T08:45:05.248710+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-20T08:45:29.544075+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.183:0/1999141339":"2024-03-21T08:46:29.591823+0000","172.21.15.183:0/2737646670":"2024-03-21T08:40:59.502103+0000","172.21.15.183:0/1814559177":"2024-03-21T08:46:29.591823+0000","172.21.15.183:0/3451815539":"2024-03-21T08:40:33.183142+0000","172.21.15.183:0/213307139":"2024-03-21T08:40:33.183142+0000","172.21.15.183:6800/2997288523":"2024-03-21T08:40:33.183142+0000","172.21.15.183:0/3285061906":"2024-03-21T08:40:33.183142+0000","172.21.15.183:6800/1361970260":"2024-03-21T08:40:59.502103+0000","172.21.15.183:6801/2997288523":"2024-03-21T08:40:33.183142+0000","172.21.15.183:6800/3255438358":"2024-03-21T08:46:29.591823+0000","172.21.15.183:6801/1361970260":"2024-03-21T08:40:59.502103+0000","172.21.15.183:0/1027467964":"2024-03-21T08:40:59.502103+0000","172.21.15.183:0/3209928172":"2024-03-21T08:40:59.502103+0000","172.21.15.183:6801/3255438358":"2024-03-21T08:46:29.591823+0000","172.21.15.183:0/3241368708":"2024-03-21T08:46:29.591823+0000","172.21.15.183:0/2903520159":"2024-03-21T08:46:29.591823+0000","172.21.15.183:0/3555591867":"2024-03-21T08:46:29.591823+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-20T08:46:31.211 INFO:journalctl@ceph.iscsi.iscsi.a.smithi194.stdout:Mar 20 08:46:30 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-iscsi-iscsi-a[47467]: debug there is no tcmu-runner data available 2024-03-20T08:46:31.888 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:46:31 smithi183 ceph-mon[25285]: mgrmap e20: y(active, since 1.02406s), standbys: x 2024-03-20T08:46:31.888 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:46:31 smithi183 ceph-mon[25285]: pgmap v3: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail 2024-03-20T08:46:31.889 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:46:31 smithi183 ceph-mon[25285]: Updating smithi194:/var/lib/ceph/4c3f3eba-e695-11ee-95c9-87774f69a715/config/ceph.client.admin.keyring 2024-03-20T08:46:31.889 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:46:31 smithi183 ceph-mon[25285]: Updating smithi183:/var/lib/ceph/4c3f3eba-e695-11ee-95c9-87774f69a715/config/ceph.client.admin.keyring 2024-03-20T08:46:31.889 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:46:31 smithi183 ceph-mon[25285]: [20/Mar/2024:08:46:30] ENGINE Bus STARTING 2024-03-20T08:46:31.889 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:46:31 smithi183 ceph-mon[25285]: [20/Mar/2024:08:46:30] ENGINE Serving on http://172.21.15.183:8765 2024-03-20T08:46:31.889 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:46:31 smithi183 ceph-mon[25285]: [20/Mar/2024:08:46:30] ENGINE Serving on https://172.21.15.183:7151 2024-03-20T08:46:31.889 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:46:31 smithi183 ceph-mon[25285]: [20/Mar/2024:08:46:30] ENGINE Bus STARTED 2024-03-20T08:46:31.889 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:46:31 smithi183 ceph-mon[25285]: from='mgr.14637 ' entity='mgr.y' 2024-03-20T08:46:31.889 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:46:31 smithi183 ceph-mon[25285]: from='mgr.14637 ' entity='mgr.y' 2024-03-20T08:46:31.889 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:46:31 smithi183 ceph-mon[25285]: from='client.14559 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-20T08:46:31.889 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:46:31 smithi183 ceph-mon[25285]: from='mgr.14637 ' entity='mgr.y' 2024-03-20T08:46:31.889 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:46:31 smithi183 ceph-mon[25285]: from='mgr.14637 ' entity='mgr.y' 2024-03-20T08:46:31.889 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:46:31 smithi183 ceph-mon[25285]: from='mgr.14637 ' entity='mgr.y' 2024-03-20T08:46:31.889 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:46:31 smithi183 ceph-mon[25285]: from='mgr.14637 ' entity='mgr.y' 2024-03-20T08:46:31.889 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:46:31 smithi183 ceph-mon[25285]: Deploying daemon alertmanager.a on smithi183 2024-03-20T08:46:31.890 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:46:31 smithi183 ceph-mon[25285]: from='client.? 172.21.15.183:0/2229900202' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-20T08:46:31.890 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:46:31 smithi183 ceph-mon[31860]: mgrmap e20: y(active, since 1.02406s), standbys: x 2024-03-20T08:46:31.890 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:46:31 smithi183 ceph-mon[31860]: pgmap v3: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail 2024-03-20T08:46:31.890 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:46:31 smithi183 ceph-mon[31860]: Updating smithi194:/var/lib/ceph/4c3f3eba-e695-11ee-95c9-87774f69a715/config/ceph.client.admin.keyring 2024-03-20T08:46:31.890 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:46:31 smithi183 ceph-mon[31860]: Updating smithi183:/var/lib/ceph/4c3f3eba-e695-11ee-95c9-87774f69a715/config/ceph.client.admin.keyring 2024-03-20T08:46:31.890 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:46:31 smithi183 ceph-mon[31860]: [20/Mar/2024:08:46:30] ENGINE Bus STARTING 2024-03-20T08:46:31.890 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:46:31 smithi183 ceph-mon[31860]: [20/Mar/2024:08:46:30] ENGINE Serving on http://172.21.15.183:8765 2024-03-20T08:46:31.890 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:46:31 smithi183 ceph-mon[31860]: [20/Mar/2024:08:46:30] ENGINE Serving on https://172.21.15.183:7151 2024-03-20T08:46:31.890 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:46:31 smithi183 ceph-mon[31860]: [20/Mar/2024:08:46:30] ENGINE Bus STARTED 2024-03-20T08:46:31.891 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:46:31 smithi183 ceph-mon[31860]: from='mgr.14637 ' entity='mgr.y' 2024-03-20T08:46:31.891 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:46:31 smithi183 ceph-mon[31860]: from='mgr.14637 ' entity='mgr.y' 2024-03-20T08:46:31.891 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:46:31 smithi183 ceph-mon[31860]: from='client.14559 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-20T08:46:31.891 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:46:31 smithi183 ceph-mon[31860]: from='mgr.14637 ' entity='mgr.y' 2024-03-20T08:46:31.891 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:46:31 smithi183 ceph-mon[31860]: from='mgr.14637 ' entity='mgr.y' 2024-03-20T08:46:31.891 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:46:31 smithi183 ceph-mon[31860]: from='mgr.14637 ' entity='mgr.y' 2024-03-20T08:46:31.891 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:46:31 smithi183 ceph-mon[31860]: from='mgr.14637 ' entity='mgr.y' 2024-03-20T08:46:31.891 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:46:31 smithi183 ceph-mon[31860]: Deploying daemon alertmanager.a on smithi183 2024-03-20T08:46:31.891 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:46:31 smithi183 ceph-mon[31860]: from='client.? 172.21.15.183:0/2229900202' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-20T08:46:31.891 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:46:31 smithi194 ceph-mon[27237]: mgrmap e20: y(active, since 1.02406s), standbys: x 2024-03-20T08:46:31.892 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:46:31 smithi194 ceph-mon[27237]: pgmap v3: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail 2024-03-20T08:46:31.892 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:46:31 smithi194 ceph-mon[27237]: Updating smithi194:/var/lib/ceph/4c3f3eba-e695-11ee-95c9-87774f69a715/config/ceph.client.admin.keyring 2024-03-20T08:46:31.892 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:46:31 smithi194 ceph-mon[27237]: Updating smithi183:/var/lib/ceph/4c3f3eba-e695-11ee-95c9-87774f69a715/config/ceph.client.admin.keyring 2024-03-20T08:46:31.892 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:46:31 smithi194 ceph-mon[27237]: [20/Mar/2024:08:46:30] ENGINE Bus STARTING 2024-03-20T08:46:31.892 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:46:31 smithi194 ceph-mon[27237]: [20/Mar/2024:08:46:30] ENGINE Serving on http://172.21.15.183:8765 2024-03-20T08:46:31.892 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:46:31 smithi194 ceph-mon[27237]: [20/Mar/2024:08:46:30] ENGINE Serving on https://172.21.15.183:7151 2024-03-20T08:46:31.892 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:46:31 smithi194 ceph-mon[27237]: [20/Mar/2024:08:46:30] ENGINE Bus STARTED 2024-03-20T08:46:31.892 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:46:31 smithi194 ceph-mon[27237]: from='mgr.14637 ' entity='mgr.y' 2024-03-20T08:46:31.892 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:46:31 smithi194 ceph-mon[27237]: from='mgr.14637 ' entity='mgr.y' 2024-03-20T08:46:31.892 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:46:31 smithi194 ceph-mon[27237]: from='client.14559 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-20T08:46:31.892 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:46:31 smithi194 ceph-mon[27237]: from='mgr.14637 ' entity='mgr.y' 2024-03-20T08:46:31.892 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:46:31 smithi194 ceph-mon[27237]: from='mgr.14637 ' entity='mgr.y' 2024-03-20T08:46:31.892 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:46:31 smithi194 ceph-mon[27237]: from='mgr.14637 ' entity='mgr.y' 2024-03-20T08:46:31.892 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:46:31 smithi194 ceph-mon[27237]: from='mgr.14637 ' entity='mgr.y' 2024-03-20T08:46:31.893 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:46:31 smithi194 ceph-mon[27237]: Deploying daemon alertmanager.a on smithi183 2024-03-20T08:46:31.893 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:46:31 smithi194 ceph-mon[27237]: from='client.? 172.21.15.183:0/2229900202' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-20T08:46:32.571 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 4c3f3eba-e695-11ee-95c9-87774f69a715 -- ceph tell osd.0 flush_pg_stats 2024-03-20T08:46:32.571 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 4c3f3eba-e695-11ee-95c9-87774f69a715 -- ceph tell osd.1 flush_pg_stats 2024-03-20T08:46:32.572 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 4c3f3eba-e695-11ee-95c9-87774f69a715 -- ceph tell osd.2 flush_pg_stats 2024-03-20T08:46:32.572 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 4c3f3eba-e695-11ee-95c9-87774f69a715 -- ceph tell osd.3 flush_pg_stats 2024-03-20T08:46:32.573 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 4c3f3eba-e695-11ee-95c9-87774f69a715 -- ceph tell osd.4 flush_pg_stats 2024-03-20T08:46:32.573 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 4c3f3eba-e695-11ee-95c9-87774f69a715 -- ceph tell osd.5 flush_pg_stats 2024-03-20T08:46:32.574 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 4c3f3eba-e695-11ee-95c9-87774f69a715 -- ceph tell osd.6 flush_pg_stats 2024-03-20T08:46:32.574 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 4c3f3eba-e695-11ee-95c9-87774f69a715 -- ceph tell osd.7 flush_pg_stats 2024-03-20T08:46:33.195 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:46:32 smithi183 ceph-mon[25285]: pgmap v4: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail 2024-03-20T08:46:33.195 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:46:32 smithi183 ceph-mon[25285]: mgrmap e21: y(active, since 2s), standbys: x 2024-03-20T08:46:33.195 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:46:32 smithi183 ceph-mon[31860]: pgmap v4: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail 2024-03-20T08:46:33.195 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:46:32 smithi183 ceph-mon[31860]: mgrmap e21: y(active, since 2s), standbys: x 2024-03-20T08:46:33.388 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:46:32 smithi194 ceph-mon[27237]: pgmap v4: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail 2024-03-20T08:46:33.389 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:46:32 smithi194 ceph-mon[27237]: mgrmap e21: y(active, since 2s), standbys: x 2024-03-20T08:46:33.415 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/4c3f3eba-e695-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T08:46:33.416 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/4c3f3eba-e695-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T08:46:33.416 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/4c3f3eba-e695-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T08:46:33.417 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/4c3f3eba-e695-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T08:46:34.317 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/4c3f3eba-e695-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T08:46:34.318 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/4c3f3eba-e695-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T08:46:34.647 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/4c3f3eba-e695-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T08:46:35.092 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:46:34 smithi183 ceph-mon[25285]: pgmap v5: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail 2024-03-20T08:46:35.092 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:46:34 smithi183 ceph-mon[25285]: mgrmap e22: y(active, since 4s), standbys: x 2024-03-20T08:46:35.092 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:46:34 smithi183 ceph-mon[31860]: pgmap v5: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail 2024-03-20T08:46:35.093 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:46:34 smithi183 ceph-mon[31860]: mgrmap e22: y(active, since 4s), standbys: x 2024-03-20T08:46:35.389 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:46:34 smithi194 ceph-mon[27237]: pgmap v5: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail 2024-03-20T08:46:35.389 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:46:34 smithi194 ceph-mon[27237]: mgrmap e22: y(active, since 4s), standbys: x 2024-03-20T08:46:37.071 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:46:36 smithi183 ceph-mon[31860]: pgmap v6: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail 2024-03-20T08:46:37.071 INFO:journalctl@ceph.mgr.y.smithi183.stdout:Mar 20 08:46:36 smithi183 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-mgr-y[25513]: ::ffff:172.21.15.194 - - [20/Mar/2024:08:46:36] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-20T08:46:37.072 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:46:36 smithi183 ceph-mon[25285]: pgmap v6: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail 2024-03-20T08:46:37.388 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:46:36 smithi194 ceph-mon[27237]: pgmap v6: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail 2024-03-20T08:46:37.884 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/4c3f3eba-e695-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T08:46:39.204 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:46:38 smithi183 ceph-mon[25285]: pgmap v7: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 30 KiB/s rd, 0 B/s wr, 17 op/s 2024-03-20T08:46:39.204 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:46:38 smithi183 ceph-mon[31860]: pgmap v7: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 30 KiB/s rd, 0 B/s wr, 17 op/s 2024-03-20T08:46:39.389 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:46:38 smithi194 ceph-mon[27237]: pgmap v7: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 30 KiB/s rd, 0 B/s wr, 17 op/s 2024-03-20T08:46:39.716 INFO:teuthology.orchestra.run.smithi183.stdout:231928233999 2024-03-20T08:46:39.717 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 4c3f3eba-e695-11ee-95c9-87774f69a715 -- ceph osd last-stat-seq osd.7 2024-03-20T08:46:41.070 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:46:40 smithi183 ceph-mon[31860]: pgmap v8: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 24 KiB/s rd, 0 B/s wr, 13 op/s 2024-03-20T08:46:41.071 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:46:40 smithi183 ceph-mon[25285]: pgmap v8: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 24 KiB/s rd, 0 B/s wr, 13 op/s 2024-03-20T08:46:41.204 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:46:40 smithi194 ceph-mon[27237]: pgmap v8: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 24 KiB/s rd, 0 B/s wr, 13 op/s 2024-03-20T08:46:41.205 INFO:journalctl@ceph.iscsi.iscsi.a.smithi194.stdout:Mar 20 08:46:40 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-iscsi-iscsi-a[47467]: debug there is no tcmu-runner data available 2024-03-20T08:46:42.309 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:46:41 smithi183 ceph-mon[31860]: from='client.14559 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-20T08:46:42.310 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:46:41 smithi183 ceph-mon[25285]: from='client.14559 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-20T08:46:42.347 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:46:41 smithi194 ceph-mon[27237]: from='client.14559 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-20T08:46:43.127 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:46:42 smithi194 ceph-mon[27237]: pgmap v9: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 20 KiB/s rd, 0 B/s wr, 11 op/s 2024-03-20T08:46:43.204 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:46:42 smithi183 ceph-mon[25285]: pgmap v9: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 20 KiB/s rd, 0 B/s wr, 11 op/s 2024-03-20T08:46:43.205 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:46:42 smithi183 ceph-mon[31860]: pgmap v9: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 20 KiB/s rd, 0 B/s wr, 11 op/s 2024-03-20T08:46:45.171 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:46:44 smithi183 ceph-mon[25285]: pgmap v10: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-03-20T08:46:45.172 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:46:44 smithi183 ceph-mon[25285]: from='mgr.14637 172.21.15.183:0/885080955' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-20T08:46:45.172 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:46:44 smithi183 ceph-mon[31860]: pgmap v10: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-03-20T08:46:45.172 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:46:44 smithi183 ceph-mon[31860]: from='mgr.14637 172.21.15.183:0/885080955' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-20T08:46:45.279 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:46:44 smithi194 ceph-mon[27237]: pgmap v10: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-03-20T08:46:45.279 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:46:44 smithi194 ceph-mon[27237]: from='mgr.14637 172.21.15.183:0/885080955' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-20T08:46:45.773 INFO:teuthology.orchestra.run.smithi183.stdout:55834574894 2024-03-20T08:46:45.774 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 4c3f3eba-e695-11ee-95c9-87774f69a715 -- ceph osd last-stat-seq osd.1 2024-03-20T08:46:47.155 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:46:46 smithi183 ceph-mon[25285]: pgmap v11: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-03-20T08:46:47.155 INFO:journalctl@ceph.mgr.y.smithi183.stdout:Mar 20 08:46:46 smithi183 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-mgr-y[25513]: ::ffff:172.21.15.194 - - [20/Mar/2024:08:46:46] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-20T08:46:47.155 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:46:46 smithi183 ceph-mon[31860]: pgmap v11: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-03-20T08:46:47.388 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:46:46 smithi194 ceph-mon[27237]: pgmap v11: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-03-20T08:46:49.204 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:46:48 smithi183 ceph-mon[25285]: pgmap v12: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-03-20T08:46:49.205 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:46:48 smithi183 ceph-mon[31860]: pgmap v12: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-03-20T08:46:49.389 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:46:48 smithi194 ceph-mon[27237]: pgmap v12: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-03-20T08:46:49.641 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/4c3f3eba-e695-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T08:46:50.013 INFO:teuthology.orchestra.run.smithi183.stdout:128849018912 2024-03-20T08:46:50.013 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 4c3f3eba-e695-11ee-95c9-87774f69a715 -- ceph osd last-stat-seq osd.4 2024-03-20T08:46:51.347 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:46:50 smithi183 ceph-mon[25285]: pgmap v13: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-20T08:46:51.347 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:46:50 smithi183 ceph-mon[31860]: pgmap v13: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-20T08:46:51.388 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:46:50 smithi194 ceph-mon[27237]: pgmap v13: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-20T08:46:51.389 INFO:journalctl@ceph.iscsi.iscsi.a.smithi194.stdout:Mar 20 08:46:50 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-iscsi-iscsi-a[47467]: debug there is no tcmu-runner data available 2024-03-20T08:46:52.138 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:46:51 smithi194 ceph-mon[27237]: from='client.14559 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-20T08:46:52.140 INFO:teuthology.orchestra.run.smithi183.stdout:197568495638 2024-03-20T08:46:52.141 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 4c3f3eba-e695-11ee-95c9-87774f69a715 -- ceph osd last-stat-seq osd.6 2024-03-20T08:46:52.454 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:46:51 smithi183 ceph-mon[25285]: from='client.14559 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-20T08:46:52.454 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:46:51 smithi183 ceph-mon[31860]: from='client.14559 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-20T08:46:52.941 INFO:teuthology.orchestra.run.smithi183.stdout:107374182438 2024-03-20T08:46:52.941 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 4c3f3eba-e695-11ee-95c9-87774f69a715 -- ceph osd last-stat-seq osd.3 2024-03-20T08:46:53.134 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:46:52 smithi183 ceph-mon[25285]: pgmap v14: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-20T08:46:53.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:46:52 smithi183 ceph-mon[31860]: pgmap v14: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-20T08:46:53.389 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:46:52 smithi194 ceph-mon[27237]: pgmap v14: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-20T08:46:53.576 INFO:teuthology.orchestra.run.smithi183.stdout:163208757274 2024-03-20T08:46:53.576 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 4c3f3eba-e695-11ee-95c9-87774f69a715 -- ceph osd last-stat-seq osd.5 2024-03-20T08:46:54.973 INFO:teuthology.orchestra.run.smithi183.stdout:231928234002 2024-03-20T08:46:55.204 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:46:54 smithi183 ceph-mon[25285]: pgmap v15: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-20T08:46:55.204 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:46:54 smithi183 ceph-mon[25285]: from='client.? 172.21.15.183:0/2024679449' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-03-20T08:46:55.205 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:46:54 smithi183 ceph-mon[31860]: pgmap v15: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-20T08:46:55.205 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:46:54 smithi183 ceph-mon[31860]: from='client.? 172.21.15.183:0/2024679449' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-03-20T08:46:55.335 INFO:teuthology.orchestra.run.smithi183.stdout:34359738422 2024-03-20T08:46:55.335 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 4c3f3eba-e695-11ee-95c9-87774f69a715 -- ceph osd last-stat-seq osd.0 2024-03-20T08:46:55.389 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:46:54 smithi194 ceph-mon[27237]: pgmap v15: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-20T08:46:55.389 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:46:54 smithi194 ceph-mon[27237]: from='client.? 172.21.15.183:0/2024679449' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-03-20T08:46:56.123 INFO:tasks.cephadm.ceph_manager.ceph:need seq 231928233999 got 231928234002 for osd.7 2024-03-20T08:46:56.123 DEBUG:teuthology.parallel:result is None 2024-03-20T08:46:57.200 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:46:57 smithi183 ceph-mon[25285]: pgmap v16: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-20T08:46:57.201 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:46:57 smithi183 ceph-mon[31860]: pgmap v16: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-20T08:46:57.201 INFO:journalctl@ceph.mgr.y.smithi183.stdout:Mar 20 08:46:56 smithi183 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-mgr-y[25513]: ::ffff:172.21.15.194 - - [20/Mar/2024:08:46:56] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-20T08:46:57.388 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:46:57 smithi194 ceph-mon[27237]: pgmap v16: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-20T08:46:58.057 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/4c3f3eba-e695-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T08:46:58.058 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/4c3f3eba-e695-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T08:46:58.106 INFO:teuthology.orchestra.run.smithi183.stdout:77309411374 2024-03-20T08:46:58.107 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 4c3f3eba-e695-11ee-95c9-87774f69a715 -- ceph osd last-stat-seq osd.2 2024-03-20T08:46:58.111 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/4c3f3eba-e695-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T08:46:58.112 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/4c3f3eba-e695-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T08:46:59.204 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:46:59 smithi183 ceph-mon[25285]: pgmap v17: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-20T08:46:59.204 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:46:59 smithi183 ceph-mon[31860]: pgmap v17: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-20T08:46:59.388 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:46:59 smithi194 ceph-mon[27237]: pgmap v17: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-20T08:47:00.204 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:47:00 smithi183 ceph-mon[25285]: from='mgr.14637 172.21.15.183:0/885080955' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-20T08:47:00.204 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:47:00 smithi183 ceph-mon[31860]: from='mgr.14637 172.21.15.183:0/885080955' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-20T08:47:00.286 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/4c3f3eba-e695-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T08:47:00.286 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/4c3f3eba-e695-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T08:47:00.388 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:47:00 smithi194 ceph-mon[27237]: from='mgr.14637 172.21.15.183:0/885080955' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-20T08:47:01.163 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:47:01 smithi183 ceph-mon[25285]: pgmap v18: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-20T08:47:01.164 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:47:01 smithi183 ceph-mon[31860]: pgmap v18: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-20T08:47:01.277 INFO:teuthology.orchestra.run.smithi183.stdout:197568495640 2024-03-20T08:47:01.388 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:47:01 smithi194 ceph-mon[27237]: pgmap v18: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-20T08:47:01.389 INFO:journalctl@ceph.iscsi.iscsi.a.smithi194.stdout:Mar 20 08:47:00 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-iscsi-iscsi-a[47467]: debug there is no tcmu-runner data available 2024-03-20T08:47:02.243 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/4c3f3eba-e695-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T08:47:02.369 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:47:02 smithi183 ceph-mon[31860]: from='client.14559 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-20T08:47:02.369 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:47:02 smithi183 ceph-mon[31860]: from='client.? 172.21.15.183:0/978729575' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-03-20T08:47:02.370 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:47:02 smithi183 ceph-mon[25285]: from='client.14559 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-20T08:47:02.370 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:47:02 smithi183 ceph-mon[25285]: from='client.? 172.21.15.183:0/978729575' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-03-20T08:47:02.388 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:47:02 smithi194 ceph-mon[27237]: from='client.14559 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-20T08:47:02.388 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:47:02 smithi194 ceph-mon[27237]: from='client.? 172.21.15.183:0/978729575' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-03-20T08:47:03.241 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:47:03 smithi194 ceph-mon[27237]: pgmap v19: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-20T08:47:03.302 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:47:03 smithi183 ceph-mon[25285]: pgmap v19: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-20T08:47:03.302 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:47:03 smithi183 ceph-mon[31860]: pgmap v19: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-20T08:47:04.125 INFO:journalctl@ceph.alertmanager.a.smithi183.stdout:Mar 20 08:47:04 smithi183 podman[59940]: 2024-03-20 08:47:04.12466494 +0000 UTC m=+3.818443126 container create 4915bdfc21cf8b2e25383805fbf6ac9bd8b63791095d9034e9f4bdc7d2a913bd (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-03-20T08:47:05.124 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:47:05 smithi183 ceph-mon[31860]: pgmap v20: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-20T08:47:05.124 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:47:05 smithi183 ceph-mon[25285]: pgmap v20: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-20T08:47:05.389 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:47:05 smithi194 ceph-mon[27237]: pgmap v20: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-20T08:47:05.411 INFO:journalctl@ceph.alertmanager.a.smithi183.stdout:Mar 20 08:47:05 smithi183 podman[59940]: 2024-03-20 08:47:05.310602592 +0000 UTC m=+5.004380749 container init 4915bdfc21cf8b2e25383805fbf6ac9bd8b63791095d9034e9f4bdc7d2a913bd (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-03-20T08:47:05.411 INFO:journalctl@ceph.alertmanager.a.smithi183.stdout:Mar 20 08:47:05 smithi183 podman[59940]: 2024-03-20 08:47:05.313761174 +0000 UTC m=+5.007539319 container start 4915bdfc21cf8b2e25383805fbf6ac9bd8b63791095d9034e9f4bdc7d2a913bd (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-03-20T08:47:05.411 INFO:journalctl@ceph.alertmanager.a.smithi183.stdout:Mar 20 08:47:05 smithi183 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-alertmanager-a[60200]: ts=2024-03-20T08:47:05.326Z caller=main.go:240 level=info msg="Starting Alertmanager" version="(version=0.25.0, branch=HEAD, revision=258fab7cdd551f2cf251ed0348f0ad7289aee789)" 2024-03-20T08:47:05.411 INFO:journalctl@ceph.alertmanager.a.smithi183.stdout:Mar 20 08:47:05 smithi183 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-alertmanager-a[60200]: ts=2024-03-20T08:47:05.327Z caller=main.go:241 level=info build_context="(go=go1.19.4, user=root@abe866dd5717, date=20221222-14:51:36)" 2024-03-20T08:47:05.411 INFO:journalctl@ceph.alertmanager.a.smithi183.stdout:Mar 20 08:47:05 smithi183 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-alertmanager-a[60200]: ts=2024-03-20T08:47:05.328Z caller=cluster.go:185 level=info component=cluster msg="setting advertise address explicitly" addr=172.21.15.183 port=9094 2024-03-20T08:47:05.411 INFO:journalctl@ceph.alertmanager.a.smithi183.stdout:Mar 20 08:47:05 smithi183 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-alertmanager-a[60200]: ts=2024-03-20T08:47:05.328Z caller=cluster.go:681 level=info component=cluster msg="Waiting for gossip to settle..." interval=2s 2024-03-20T08:47:05.411 INFO:journalctl@ceph.alertmanager.a.smithi183.stdout:Mar 20 08:47:05 smithi183 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-alertmanager-a[60200]: ts=2024-03-20T08:47:05.353Z caller=coordinator.go:113 level=info component=configuration msg="Loading configuration file" file=/etc/alertmanager/alertmanager.yml 2024-03-20T08:47:05.411 INFO:journalctl@ceph.alertmanager.a.smithi183.stdout:Mar 20 08:47:05 smithi183 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-alertmanager-a[60200]: ts=2024-03-20T08:47:05.353Z caller=coordinator.go:126 level=info component=configuration msg="Completed loading of configuration file" file=/etc/alertmanager/alertmanager.yml 2024-03-20T08:47:05.411 INFO:journalctl@ceph.alertmanager.a.smithi183.stdout:Mar 20 08:47:05 smithi183 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-alertmanager-a[60200]: ts=2024-03-20T08:47:05.356Z caller=tls_config.go:232 level=info msg="Listening on" address=[::]:9093 2024-03-20T08:47:05.411 INFO:journalctl@ceph.alertmanager.a.smithi183.stdout:Mar 20 08:47:05 smithi183 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-alertmanager-a[60200]: ts=2024-03-20T08:47:05.356Z caller=tls_config.go:235 level=info msg="TLS is disabled." http2=false address=[::]:9093 2024-03-20T08:47:05.411 INFO:journalctl@ceph.alertmanager.a.smithi183.stdout:Mar 20 08:47:05 smithi183 bash[59940]: 4915bdfc21cf8b2e25383805fbf6ac9bd8b63791095d9034e9f4bdc7d2a913bd 2024-03-20T08:47:05.704 INFO:journalctl@ceph.mgr.y.smithi183.stdout:Mar 20 08:47:05 smithi183 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-mgr-y[25513]: [20/Mar/2024:08:47:05] ENGINE Bus STOPPING 2024-03-20T08:47:05.705 INFO:journalctl@ceph.mgr.y.smithi183.stdout:Mar 20 08:47:05 smithi183 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-mgr-y[25513]: [20/Mar/2024:08:47:05] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-03-20T08:47:05.705 INFO:journalctl@ceph.mgr.y.smithi183.stdout:Mar 20 08:47:05 smithi183 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-mgr-y[25513]: [20/Mar/2024:08:47:05] ENGINE Bus STOPPED 2024-03-20T08:47:05.705 INFO:journalctl@ceph.mgr.y.smithi183.stdout:Mar 20 08:47:05 smithi183 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-mgr-y[25513]: [20/Mar/2024:08:47:05] ENGINE Bus STARTING 2024-03-20T08:47:05.705 INFO:journalctl@ceph.alertmanager.a.smithi183.stdout:Mar 20 08:47:05 smithi183 systemd[1]: Started Ceph alertmanager.a for 4c3f3eba-e695-11ee-95c9-87774f69a715. 2024-03-20T08:47:05.840 INFO:teuthology.orchestra.run.smithi183.stdout:128849018915 2024-03-20T08:47:05.961 INFO:journalctl@ceph.mgr.y.smithi183.stdout:Mar 20 08:47:05 smithi183 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-mgr-y[25513]: [20/Mar/2024:08:47:05] ENGINE Serving on http://:::9283 2024-03-20T08:47:05.961 INFO:journalctl@ceph.mgr.y.smithi183.stdout:Mar 20 08:47:05 smithi183 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-mgr-y[25513]: [20/Mar/2024:08:47:05] ENGINE Bus STARTED 2024-03-20T08:47:06.429 INFO:teuthology.orchestra.run.smithi183.stdout:34359738425 2024-03-20T08:47:06.639 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:47:06 smithi194 ceph-mon[27237]: from='mgr.14637 ' entity='mgr.y' 2024-03-20T08:47:06.639 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:47:06 smithi194 ceph-mon[27237]: from='mgr.14637 ' entity='mgr.y' 2024-03-20T08:47:06.639 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:47:06 smithi194 ceph-mon[27237]: from='mgr.14637 ' entity='mgr.y' 2024-03-20T08:47:06.639 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:47:06 smithi194 ceph-mon[27237]: from='mgr.14637 ' entity='mgr.y' 2024-03-20T08:47:06.639 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:47:06 smithi194 ceph-mon[27237]: from='mgr.14637 ' entity='mgr.y' 2024-03-20T08:47:06.639 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:47:06 smithi194 ceph-mon[27237]: from='mgr.14637 ' entity='mgr.y' 2024-03-20T08:47:06.639 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:47:06 smithi194 ceph-mon[27237]: from='mgr.14637 172.21.15.183:0/885080955' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-03-20T08:47:06.639 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:47:06 smithi194 ceph-mon[27237]: from='mgr.14637 ' entity='mgr.y' 2024-03-20T08:47:06.639 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:47:06 smithi194 ceph-mon[27237]: from='client.? 172.21.15.183:0/4254900184' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-03-20T08:47:06.639 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:47:06 smithi194 ceph-mon[27237]: from='client.? 172.21.15.183:0/2125587194' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-03-20T08:47:06.654 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:47:06 smithi183 ceph-mon[25285]: from='mgr.14637 ' entity='mgr.y' 2024-03-20T08:47:06.654 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:47:06 smithi183 ceph-mon[25285]: from='mgr.14637 ' entity='mgr.y' 2024-03-20T08:47:06.654 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:47:06 smithi183 ceph-mon[25285]: from='mgr.14637 ' entity='mgr.y' 2024-03-20T08:47:06.654 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:47:06 smithi183 ceph-mon[25285]: from='mgr.14637 ' entity='mgr.y' 2024-03-20T08:47:06.654 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:47:06 smithi183 ceph-mon[25285]: from='mgr.14637 ' entity='mgr.y' 2024-03-20T08:47:06.654 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:47:06 smithi183 ceph-mon[25285]: from='mgr.14637 ' entity='mgr.y' 2024-03-20T08:47:06.655 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:47:06 smithi183 ceph-mon[25285]: from='mgr.14637 172.21.15.183:0/885080955' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-03-20T08:47:06.655 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:47:06 smithi183 ceph-mon[25285]: from='mgr.14637 ' entity='mgr.y' 2024-03-20T08:47:06.655 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:47:06 smithi183 ceph-mon[25285]: from='client.? 172.21.15.183:0/4254900184' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-03-20T08:47:06.655 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:47:06 smithi183 ceph-mon[25285]: from='client.? 172.21.15.183:0/2125587194' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-03-20T08:47:06.655 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:47:06 smithi183 ceph-mon[31860]: from='mgr.14637 ' entity='mgr.y' 2024-03-20T08:47:06.655 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:47:06 smithi183 ceph-mon[31860]: from='mgr.14637 ' entity='mgr.y' 2024-03-20T08:47:06.655 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:47:06 smithi183 ceph-mon[31860]: from='mgr.14637 ' entity='mgr.y' 2024-03-20T08:47:06.655 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:47:06 smithi183 ceph-mon[31860]: from='mgr.14637 ' entity='mgr.y' 2024-03-20T08:47:06.655 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:47:06 smithi183 ceph-mon[31860]: from='mgr.14637 ' entity='mgr.y' 2024-03-20T08:47:06.655 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:47:06 smithi183 ceph-mon[31860]: from='mgr.14637 ' entity='mgr.y' 2024-03-20T08:47:06.655 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:47:06 smithi183 ceph-mon[31860]: from='mgr.14637 172.21.15.183:0/885080955' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-03-20T08:47:06.656 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:47:06 smithi183 ceph-mon[31860]: from='mgr.14637 ' entity='mgr.y' 2024-03-20T08:47:06.656 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:47:06 smithi183 ceph-mon[31860]: from='client.? 172.21.15.183:0/4254900184' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-03-20T08:47:06.656 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:47:06 smithi183 ceph-mon[31860]: from='client.? 172.21.15.183:0/2125587194' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-03-20T08:47:06.680 INFO:tasks.cephadm.ceph_manager.ceph:need seq 128849018912 got 128849018915 for osd.4 2024-03-20T08:47:06.681 DEBUG:teuthology.parallel:result is None 2024-03-20T08:47:07.204 INFO:journalctl@ceph.mgr.y.smithi183.stdout:Mar 20 08:47:06 smithi183 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-mgr-y[25513]: ::ffff:172.21.15.194 - - [20/Mar/2024:08:47:06] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-20T08:47:07.593 INFO:journalctl@ceph.alertmanager.a.smithi183.stdout:Mar 20 08:47:07 smithi183 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-alertmanager-a[60200]: ts=2024-03-20T08:47:07.329Z caller=cluster.go:706 level=info component=cluster msg="gossip not settled" polls=0 before=0 now=1 elapsed=2.000170364s 2024-03-20T08:47:07.594 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:47:07 smithi183 ceph-mon[31860]: Regenerating cephadm self-signed grafana TLS certificates 2024-03-20T08:47:07.594 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:47:07 smithi183 ceph-mon[31860]: pgmap v21: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-20T08:47:07.594 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:47:07 smithi183 ceph-mon[31860]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-03-20T08:47:07.594 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:47:07 smithi183 ceph-mon[31860]: Deploying daemon grafana.a on smithi194 2024-03-20T08:47:07.594 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:47:07 smithi183 ceph-mon[25285]: Regenerating cephadm self-signed grafana TLS certificates 2024-03-20T08:47:07.594 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:47:07 smithi183 ceph-mon[25285]: pgmap v21: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-20T08:47:07.594 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:47:07 smithi183 ceph-mon[25285]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-03-20T08:47:07.594 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:47:07 smithi183 ceph-mon[25285]: Deploying daemon grafana.a on smithi194 2024-03-20T08:47:07.638 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:47:07 smithi194 ceph-mon[27237]: Regenerating cephadm self-signed grafana TLS certificates 2024-03-20T08:47:07.639 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:47:07 smithi194 ceph-mon[27237]: pgmap v21: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-20T08:47:07.639 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:47:07 smithi194 ceph-mon[27237]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-03-20T08:47:07.639 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:47:07 smithi194 ceph-mon[27237]: Deploying daemon grafana.a on smithi194 2024-03-20T08:47:08.792 INFO:teuthology.orchestra.run.smithi183.stdout:163208757279 2024-03-20T08:47:09.218 INFO:tasks.cephadm.ceph_manager.ceph:need seq 197568495638 got 197568495640 for osd.6 2024-03-20T08:47:09.218 DEBUG:teuthology.parallel:result is None 2024-03-20T08:47:09.659 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:47:09 smithi183 ceph-mon[25285]: pgmap v22: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-20T08:47:09.660 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:47:09 smithi183 ceph-mon[25285]: from='client.? 172.21.15.183:0/1890453206' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-03-20T08:47:09.660 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:47:09 smithi183 ceph-mon[31860]: pgmap v22: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-20T08:47:09.660 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:47:09 smithi183 ceph-mon[31860]: from='client.? 172.21.15.183:0/1890453206' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-03-20T08:47:09.888 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:47:09 smithi194 ceph-mon[27237]: pgmap v22: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-20T08:47:09.888 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:47:09 smithi194 ceph-mon[27237]: from='client.? 172.21.15.183:0/1890453206' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-03-20T08:47:10.083 INFO:tasks.cephadm.ceph_manager.ceph:need seq 34359738422 got 34359738425 for osd.0 2024-03-20T08:47:10.083 DEBUG:teuthology.parallel:result is None 2024-03-20T08:47:10.954 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:47:10 smithi183 ceph-mon[25285]: pgmap v23: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-20T08:47:10.954 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:47:10 smithi183 ceph-mon[25285]: from='mgr.14637 ' entity='mgr.y' 2024-03-20T08:47:10.955 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:47:10 smithi183 ceph-mon[31860]: pgmap v23: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-20T08:47:10.955 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:47:10 smithi183 ceph-mon[31860]: from='mgr.14637 ' entity='mgr.y' 2024-03-20T08:47:10.977 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:47:10 smithi194 ceph-mon[27237]: pgmap v23: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-20T08:47:10.978 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:47:10 smithi194 ceph-mon[27237]: from='mgr.14637 ' entity='mgr.y' 2024-03-20T08:47:11.144 INFO:teuthology.orchestra.run.smithi183.stdout:55834574900 2024-03-20T08:47:11.388 INFO:journalctl@ceph.iscsi.iscsi.a.smithi194.stdout:Mar 20 08:47:10 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-iscsi-iscsi-a[47467]: debug there is no tcmu-runner data available 2024-03-20T08:47:11.799 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:47:11 smithi183 ceph-mon[25285]: from='client.14559 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-20T08:47:11.800 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:47:11 smithi183 ceph-mon[25285]: from='client.? 172.21.15.183:0/162123835' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-03-20T08:47:11.800 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:47:11 smithi183 ceph-mon[31860]: from='client.14559 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-20T08:47:11.800 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:47:11 smithi183 ceph-mon[31860]: from='client.? 172.21.15.183:0/162123835' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-03-20T08:47:12.066 INFO:tasks.cephadm.ceph_manager.ceph:need seq 163208757274 got 163208757279 for osd.5 2024-03-20T08:47:12.066 DEBUG:teuthology.parallel:result is None 2024-03-20T08:47:12.138 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:47:11 smithi194 ceph-mon[27237]: from='client.14559 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-20T08:47:12.138 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:47:11 smithi194 ceph-mon[27237]: from='client.? 172.21.15.183:0/162123835' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-03-20T08:47:12.723 INFO:teuthology.orchestra.run.smithi183.stdout:107374182443 2024-03-20T08:47:12.917 INFO:tasks.cephadm.ceph_manager.ceph:need seq 55834574894 got 55834574900 for osd.1 2024-03-20T08:47:12.918 DEBUG:teuthology.parallel:result is None 2024-03-20T08:47:12.954 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:47:12 smithi183 ceph-mon[25285]: pgmap v24: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-20T08:47:12.954 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:47:12 smithi183 ceph-mon[31860]: pgmap v24: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-20T08:47:13.138 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:47:12 smithi194 ceph-mon[27237]: pgmap v24: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-20T08:47:13.466 INFO:teuthology.orchestra.run.smithi183.stdout:77309411377 2024-03-20T08:47:13.954 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:47:13 smithi183 ceph-mon[25285]: from='client.? 172.21.15.183:0/1217115211' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-03-20T08:47:13.954 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:47:13 smithi183 ceph-mon[25285]: from='client.? 172.21.15.183:0/1153948962' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-03-20T08:47:13.954 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:47:13 smithi183 ceph-mon[31860]: from='client.? 172.21.15.183:0/1217115211' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-03-20T08:47:13.955 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:47:13 smithi183 ceph-mon[31860]: from='client.? 172.21.15.183:0/1153948962' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-03-20T08:47:14.138 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:47:13 smithi194 ceph-mon[27237]: from='client.? 172.21.15.183:0/1217115211' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-03-20T08:47:14.139 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:47:13 smithi194 ceph-mon[27237]: from='client.? 172.21.15.183:0/1153948962' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-03-20T08:47:14.332 INFO:tasks.cephadm.ceph_manager.ceph:need seq 107374182438 got 107374182443 for osd.3 2024-03-20T08:47:14.333 DEBUG:teuthology.parallel:result is None 2024-03-20T08:47:15.026 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:47:14 smithi183 ceph-mon[25285]: pgmap v25: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-20T08:47:15.027 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:47:14 smithi183 ceph-mon[31860]: pgmap v25: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-20T08:47:15.053 INFO:tasks.cephadm.ceph_manager.ceph:need seq 77309411374 got 77309411377 for osd.2 2024-03-20T08:47:15.054 DEBUG:teuthology.parallel:result is None 2024-03-20T08:47:15.054 INFO:tasks.cephadm.ceph_manager.ceph:waiting for clean 2024-03-20T08:47:15.054 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 4c3f3eba-e695-11ee-95c9-87774f69a715 -- ceph pg dump --format=json 2024-03-20T08:47:15.138 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:47:14 smithi194 ceph-mon[27237]: pgmap v25: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-20T08:47:15.597 INFO:journalctl@ceph.alertmanager.a.smithi183.stdout:Mar 20 08:47:15 smithi183 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-alertmanager-a[60200]: ts=2024-03-20T08:47:15.330Z caller=cluster.go:698 level=info component=cluster msg="gossip settled; proceeding" elapsed=10.001968013s 2024-03-20T08:47:15.924 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:47:15 smithi183 ceph-mon[25285]: from='mgr.14637 172.21.15.183:0/885080955' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-20T08:47:15.925 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:47:15 smithi183 ceph-mon[31860]: from='mgr.14637 172.21.15.183:0/885080955' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-20T08:47:15.938 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/4c3f3eba-e695-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T08:47:16.138 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:47:15 smithi194 ceph-mon[27237]: from='mgr.14637 172.21.15.183:0/885080955' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-20T08:47:16.940 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:47:16 smithi183 ceph-mon[25285]: pgmap v26: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-20T08:47:16.940 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:47:16 smithi183 ceph-mon[31860]: pgmap v26: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-20T08:47:17.138 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:47:16 smithi194 ceph-mon[27237]: pgmap v26: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-20T08:47:17.204 INFO:journalctl@ceph.mgr.y.smithi183.stdout:Mar 20 08:47:16 smithi183 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-mgr-y[25513]: ::ffff:172.21.15.194 - - [20/Mar/2024:08:47:16] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-20T08:47:18.775 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:47:18 smithi183 ceph-mon[25285]: pgmap v27: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-20T08:47:18.776 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:47:18 smithi183 ceph-mon[31860]: pgmap v27: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-20T08:47:19.096 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-20T08:47:19.099 INFO:teuthology.orchestra.run.smithi183.stderr:dumped all 2024-03-20T08:47:19.139 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:47:18 smithi194 ceph-mon[27237]: pgmap v27: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-20T08:47:19.888 INFO:teuthology.orchestra.run.smithi183.stdout:{"pg_ready":true,"pg_map":{"version":27,"stamp":"2024-03-20T08:47:17.628373+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":596392,"num_objects":198,"num_object_clones":0,"num_object_copies":594,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":198,"num_whiteouts":0,"num_read":1060,"num_read_kb":903,"num_write":681,"num_write_kb":4027,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":686,"ondisk_log_size":686,"up":396,"acting":396,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":396,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749928448,"kb_used":244752,"kb_used_data":7804,"kb_used_omap":12,"kb_used_meta":236851,"kb_avail":749683696,"statfs":{"total":767926730752,"available":767676104704,"internally_reserved":0,"allocated":7991296,"data_stored":4314008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":242535984},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002758"},"pg_stats":[{"pgid":"6.1a","version":"0'0","reported_seq":17,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-20T08:46:29.658663+0000","last_change":"2024-03-20T08:45:55.683275+0000","last_active":"2024-03-20T08:46:29.658663+0000","last_peered":"2024-03-20T08:46:29.658663+0000","last_clean":"2024-03-20T08:46:29.658663+0000","last_became_active":"2024-03-20T08:45:55.682979+0000","last_became_peered":"2024-03-20T08:45:55.682979+0000","last_unstale":"2024-03-20T08:46:29.658663+0000","last_undegraded":"2024-03-20T08:46:29.658663+0000","last_fullsized":"2024-03-20T08:46:29.658663+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-20T08:45:54.654511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:45:54.654511+0000","last_clean_scrub_stamp":"2024-03-20T08:45:54.654511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:58:05.609841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":21,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-20T08:46:29.605562+0000","last_change":"2024-03-20T08:45:53.657515+0000","last_active":"2024-03-20T08:46:29.605562+0000","last_peered":"2024-03-20T08:46:29.605562+0000","last_clean":"2024-03-20T08:46:29.605562+0000","last_became_active":"2024-03-20T08:45:53.657201+0000","last_became_peered":"2024-03-20T08:45:53.657201+0000","last_unstale":"2024-03-20T08:46:29.605562+0000","last_undegraded":"2024-03-20T08:46:29.605562+0000","last_fullsized":"2024-03-20T08:46:29.605562+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-20T08:45:52.630650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:45:52.630650+0000","last_clean_scrub_stamp":"2024-03-20T08:45:52.630650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T18:04:36.348891+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":29,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-20T08:46:29.611418+0000","last_change":"2024-03-20T08:45:49.627677+0000","last_active":"2024-03-20T08:46:29.611418+0000","last_peered":"2024-03-20T08:46:29.611418+0000","last_clean":"2024-03-20T08:46:29.611418+0000","last_became_active":"2024-03-20T08:45:49.627383+0000","last_became_peered":"2024-03-20T08:45:49.627383+0000","last_unstale":"2024-03-20T08:46:29.611418+0000","last_undegraded":"2024-03-20T08:46:29.611418+0000","last_fullsized":"2024-03-20T08:46:29.611418+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-20T08:45:48.603868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:45:48.603868+0000","last_clean_scrub_stamp":"2024-03-20T08:45:48.603868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:27:10.134498+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"67'9","reported_seq":41,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-20T08:46:29.658539+0000","last_change":"2024-03-20T08:45:51.652872+0000","last_active":"2024-03-20T08:46:29.658539+0000","last_peered":"2024-03-20T08:46:29.658539+0000","last_clean":"2024-03-20T08:46:29.658539+0000","last_became_active":"2024-03-20T08:45:51.652583+0000","last_became_peered":"2024-03-20T08:45:51.652583+0000","last_unstale":"2024-03-20T08:46:29.658539+0000","last_undegraded":"2024-03-20T08:46:29.658539+0000","last_fullsized":"2024-03-20T08:46:29.658539+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-20T08:45:50.616760+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:45:50.616760+0000","last_clean_scrub_stamp":"2024-03-20T08:45:50.616760+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T19:08:28.230482+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":17,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-20T08:46:29.686849+0000","last_change":"2024-03-20T08:45:55.816915+0000","last_active":"2024-03-20T08:46:29.686849+0000","last_peered":"2024-03-20T08:46:29.686849+0000","last_clean":"2024-03-20T08:46:29.686849+0000","last_became_active":"2024-03-20T08:45:55.816614+0000","last_became_peered":"2024-03-20T08:45:55.816614+0000","last_unstale":"2024-03-20T08:46:29.686849+0000","last_undegraded":"2024-03-20T08:46:29.686849+0000","last_fullsized":"2024-03-20T08:46:29.686849+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-20T08:45:54.654511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:45:54.654511+0000","last_clean_scrub_stamp":"2024-03-20T08:45:54.654511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T15:47:20.447708+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":21,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-20T08:46:29.658586+0000","last_change":"2024-03-20T08:45:53.657954+0000","last_active":"2024-03-20T08:46:29.658586+0000","last_peered":"2024-03-20T08:46:29.658586+0000","last_clean":"2024-03-20T08:46:29.658586+0000","last_became_active":"2024-03-20T08:45:53.657651+0000","last_became_peered":"2024-03-20T08:45:53.657651+0000","last_unstale":"2024-03-20T08:46:29.658586+0000","last_undegraded":"2024-03-20T08:46:29.658586+0000","last_fullsized":"2024-03-20T08:46:29.658586+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-20T08:45:52.630650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:45:52.630650+0000","last_clean_scrub_stamp":"2024-03-20T08:45:52.630650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T12:46:14.847461+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":29,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-20T08:46:29.687374+0000","last_change":"2024-03-20T08:45:49.628867+0000","last_active":"2024-03-20T08:46:29.687374+0000","last_peered":"2024-03-20T08:46:29.687374+0000","last_clean":"2024-03-20T08:46:29.687374+0000","last_became_active":"2024-03-20T08:45:49.628599+0000","last_became_peered":"2024-03-20T08:45:49.628599+0000","last_unstale":"2024-03-20T08:46:29.687374+0000","last_undegraded":"2024-03-20T08:46:29.687374+0000","last_fullsized":"2024-03-20T08:46:29.687374+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-20T08:45:48.603868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:45:48.603868+0000","last_clean_scrub_stamp":"2024-03-20T08:45:48.603868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T10:18:06.481115+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"67'15","reported_seq":50,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-20T08:46:29.685214+0000","last_change":"2024-03-20T08:45:51.646981+0000","last_active":"2024-03-20T08:46:29.685214+0000","last_peered":"2024-03-20T08:46:29.685214+0000","last_clean":"2024-03-20T08:46:29.685214+0000","last_became_active":"2024-03-20T08:45:51.646639+0000","last_became_peered":"2024-03-20T08:45:51.646639+0000","last_unstale":"2024-03-20T08:46:29.685214+0000","last_undegraded":"2024-03-20T08:46:29.685214+0000","last_fullsized":"2024-03-20T08:46:29.685214+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-20T08:45:50.616760+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:45:50.616760+0000","last_clean_scrub_stamp":"2024-03-20T08:45:50.616760+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T10:10:37.794721+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":17,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-20T08:46:29.609925+0000","last_change":"2024-03-20T08:45:55.682917+0000","last_active":"2024-03-20T08:46:29.609925+0000","last_peered":"2024-03-20T08:46:29.609925+0000","last_clean":"2024-03-20T08:46:29.609925+0000","last_became_active":"2024-03-20T08:45:55.682590+0000","last_became_peered":"2024-03-20T08:45:55.682590+0000","last_unstale":"2024-03-20T08:46:29.609925+0000","last_undegraded":"2024-03-20T08:46:29.609925+0000","last_fullsized":"2024-03-20T08:46:29.609925+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-20T08:45:54.654511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:45:54.654511+0000","last_clean_scrub_stamp":"2024-03-20T08:45:54.654511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:46:35.399856+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":21,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-20T08:46:29.607140+0000","last_change":"2024-03-20T08:45:53.657201+0000","last_active":"2024-03-20T08:46:29.607140+0000","last_peered":"2024-03-20T08:46:29.607140+0000","last_clean":"2024-03-20T08:46:29.607140+0000","last_became_active":"2024-03-20T08:45:53.656878+0000","last_became_peered":"2024-03-20T08:45:53.656878+0000","last_unstale":"2024-03-20T08:46:29.607140+0000","last_undegraded":"2024-03-20T08:46:29.607140+0000","last_fullsized":"2024-03-20T08:46:29.607140+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-20T08:45:52.630650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:45:52.630650+0000","last_clean_scrub_stamp":"2024-03-20T08:45:52.630650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:15:20.079183+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":29,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-20T08:46:29.606894+0000","last_change":"2024-03-20T08:45:49.631062+0000","last_active":"2024-03-20T08:46:29.606894+0000","last_peered":"2024-03-20T08:46:29.606894+0000","last_clean":"2024-03-20T08:46:29.606894+0000","last_became_active":"2024-03-20T08:45:49.630704+0000","last_became_peered":"2024-03-20T08:45:49.630704+0000","last_unstale":"2024-03-20T08:46:29.606894+0000","last_undegraded":"2024-03-20T08:46:29.606894+0000","last_fullsized":"2024-03-20T08:46:29.606894+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-20T08:45:48.603868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:45:48.603868+0000","last_clean_scrub_stamp":"2024-03-20T08:45:48.603868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T10:19:31.295642+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"67'9","reported_seq":41,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-20T08:46:29.658880+0000","last_change":"2024-03-20T08:45:51.646977+0000","last_active":"2024-03-20T08:46:29.658880+0000","last_peered":"2024-03-20T08:46:29.658880+0000","last_clean":"2024-03-20T08:46:29.658880+0000","last_became_active":"2024-03-20T08:45:51.646704+0000","last_became_peered":"2024-03-20T08:45:51.646704+0000","last_unstale":"2024-03-20T08:46:29.658880+0000","last_undegraded":"2024-03-20T08:46:29.658880+0000","last_fullsized":"2024-03-20T08:46:29.658880+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-20T08:45:50.616760+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:45:50.616760+0000","last_clean_scrub_stamp":"2024-03-20T08:45:50.616760+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T11:15:44.978821+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":17,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-20T08:46:29.606541+0000","last_change":"2024-03-20T08:45:55.689614+0000","last_active":"2024-03-20T08:46:29.606541+0000","last_peered":"2024-03-20T08:46:29.606541+0000","last_clean":"2024-03-20T08:46:29.606541+0000","last_became_active":"2024-03-20T08:45:55.689197+0000","last_became_peered":"2024-03-20T08:45:55.689197+0000","last_unstale":"2024-03-20T08:46:29.606541+0000","last_undegraded":"2024-03-20T08:46:29.606541+0000","last_fullsized":"2024-03-20T08:46:29.606541+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-20T08:45:54.654511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:45:54.654511+0000","last_clean_scrub_stamp":"2024-03-20T08:45:54.654511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T18:52:31.062423+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":21,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-20T08:46:29.683639+0000","last_change":"2024-03-20T08:45:53.664699+0000","last_active":"2024-03-20T08:46:29.683639+0000","last_peered":"2024-03-20T08:46:29.683639+0000","last_clean":"2024-03-20T08:46:29.683639+0000","last_became_active":"2024-03-20T08:45:53.663369+0000","last_became_peered":"2024-03-20T08:45:53.663369+0000","last_unstale":"2024-03-20T08:46:29.683639+0000","last_undegraded":"2024-03-20T08:46:29.683639+0000","last_fullsized":"2024-03-20T08:46:29.683639+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-20T08:45:52.630650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:45:52.630650+0000","last_clean_scrub_stamp":"2024-03-20T08:45:52.630650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T18:48:08.339895+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":29,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-20T08:46:29.607202+0000","last_change":"2024-03-20T08:45:49.627016+0000","last_active":"2024-03-20T08:46:29.607202+0000","last_peered":"2024-03-20T08:46:29.607202+0000","last_clean":"2024-03-20T08:46:29.607202+0000","last_became_active":"2024-03-20T08:45:49.626685+0000","last_became_peered":"2024-03-20T08:45:49.626685+0000","last_unstale":"2024-03-20T08:46:29.607202+0000","last_undegraded":"2024-03-20T08:46:29.607202+0000","last_fullsized":"2024-03-20T08:46:29.607202+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-20T08:45:48.603868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:45:48.603868+0000","last_clean_scrub_stamp":"2024-03-20T08:45:48.603868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T09:29:08.243782+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"67'5","reported_seq":35,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-20T08:46:29.658912+0000","last_change":"2024-03-20T08:45:51.652825+0000","last_active":"2024-03-20T08:46:29.658912+0000","last_peered":"2024-03-20T08:46:29.658912+0000","last_clean":"2024-03-20T08:46:29.658912+0000","last_became_active":"2024-03-20T08:45:51.652540+0000","last_became_peered":"2024-03-20T08:45:51.652540+0000","last_unstale":"2024-03-20T08:46:29.658912+0000","last_undegraded":"2024-03-20T08:46:29.658912+0000","last_fullsized":"2024-03-20T08:46:29.658912+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-20T08:45:50.616760+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:45:50.616760+0000","last_clean_scrub_stamp":"2024-03-20T08:45:50.616760+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T10:30:20.738617+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":7,"num_write":6,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":17,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-20T08:46:29.659315+0000","last_change":"2024-03-20T08:45:55.814816+0000","last_active":"2024-03-20T08:46:29.659315+0000","last_peered":"2024-03-20T08:46:29.659315+0000","last_clean":"2024-03-20T08:46:29.659315+0000","last_became_active":"2024-03-20T08:45:55.814511+0000","last_became_peered":"2024-03-20T08:45:55.814511+0000","last_unstale":"2024-03-20T08:46:29.659315+0000","last_undegraded":"2024-03-20T08:46:29.659315+0000","last_fullsized":"2024-03-20T08:46:29.659315+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-20T08:45:54.654511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:45:54.654511+0000","last_clean_scrub_stamp":"2024-03-20T08:45:54.654511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T11:57:19.915113+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":21,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-20T08:46:29.606987+0000","last_change":"2024-03-20T08:45:53.662681+0000","last_active":"2024-03-20T08:46:29.606987+0000","last_peered":"2024-03-20T08:46:29.606987+0000","last_clean":"2024-03-20T08:46:29.606987+0000","last_became_active":"2024-03-20T08:45:53.662060+0000","last_became_peered":"2024-03-20T08:45:53.662060+0000","last_unstale":"2024-03-20T08:46:29.606987+0000","last_undegraded":"2024-03-20T08:46:29.606987+0000","last_fullsized":"2024-03-20T08:46:29.606987+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-20T08:45:52.630650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:45:52.630650+0000","last_clean_scrub_stamp":"2024-03-20T08:45:52.630650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T17:04:40.358859+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":29,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-20T08:46:29.611069+0000","last_change":"2024-03-20T08:45:49.629580+0000","last_active":"2024-03-20T08:46:29.611069+0000","last_peered":"2024-03-20T08:46:29.611069+0000","last_clean":"2024-03-20T08:46:29.611069+0000","last_became_active":"2024-03-20T08:45:49.629271+0000","last_became_peered":"2024-03-20T08:45:49.629271+0000","last_unstale":"2024-03-20T08:46:29.611069+0000","last_undegraded":"2024-03-20T08:46:29.611069+0000","last_fullsized":"2024-03-20T08:46:29.611069+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-20T08:45:48.603868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:45:48.603868+0000","last_clean_scrub_stamp":"2024-03-20T08:45:48.603868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T20:16:47.586756+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"67'15","reported_seq":50,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-20T08:46:29.684270+0000","last_change":"2024-03-20T08:45:51.644824+0000","last_active":"2024-03-20T08:46:29.684270+0000","last_peered":"2024-03-20T08:46:29.684270+0000","last_clean":"2024-03-20T08:46:29.684270+0000","last_became_active":"2024-03-20T08:45:51.644533+0000","last_became_peered":"2024-03-20T08:45:51.644533+0000","last_unstale":"2024-03-20T08:46:29.684270+0000","last_undegraded":"2024-03-20T08:46:29.684270+0000","last_fullsized":"2024-03-20T08:46:29.684270+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-20T08:45:50.616760+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:45:50.616760+0000","last_clean_scrub_stamp":"2024-03-20T08:45:50.616760+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T12:56:51.465613+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":17,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-20T08:46:29.687264+0000","last_change":"2024-03-20T08:45:55.815272+0000","last_active":"2024-03-20T08:46:29.687264+0000","last_peered":"2024-03-20T08:46:29.687264+0000","last_clean":"2024-03-20T08:46:29.687264+0000","last_became_active":"2024-03-20T08:45:55.814811+0000","last_became_peered":"2024-03-20T08:45:55.814811+0000","last_unstale":"2024-03-20T08:46:29.687264+0000","last_undegraded":"2024-03-20T08:46:29.687264+0000","last_fullsized":"2024-03-20T08:46:29.687264+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-20T08:45:54.654511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:45:54.654511+0000","last_clean_scrub_stamp":"2024-03-20T08:45:54.654511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T15:07:10.829897+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":21,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-20T08:46:29.659062+0000","last_change":"2024-03-20T08:45:53.661844+0000","last_active":"2024-03-20T08:46:29.659062+0000","last_peered":"2024-03-20T08:46:29.659062+0000","last_clean":"2024-03-20T08:46:29.659062+0000","last_became_active":"2024-03-20T08:45:53.661546+0000","last_became_peered":"2024-03-20T08:45:53.661546+0000","last_unstale":"2024-03-20T08:46:29.659062+0000","last_undegraded":"2024-03-20T08:46:29.659062+0000","last_fullsized":"2024-03-20T08:46:29.659062+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-20T08:45:52.630650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:45:52.630650+0000","last_clean_scrub_stamp":"2024-03-20T08:45:52.630650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T18:20:17.789450+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":29,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-20T08:46:29.658281+0000","last_change":"2024-03-20T08:45:49.624287+0000","last_active":"2024-03-20T08:46:29.658281+0000","last_peered":"2024-03-20T08:46:29.658281+0000","last_clean":"2024-03-20T08:46:29.658281+0000","last_became_active":"2024-03-20T08:45:49.622798+0000","last_became_peered":"2024-03-20T08:45:49.622798+0000","last_unstale":"2024-03-20T08:46:29.658281+0000","last_undegraded":"2024-03-20T08:46:29.658281+0000","last_fullsized":"2024-03-20T08:46:29.658281+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-20T08:45:48.603868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:45:48.603868+0000","last_clean_scrub_stamp":"2024-03-20T08:45:48.603868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T11:45:14.127073+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"67'12","reported_seq":48,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-20T08:46:29.685645+0000","last_change":"2024-03-20T08:45:51.647664+0000","last_active":"2024-03-20T08:46:29.685645+0000","last_peered":"2024-03-20T08:46:29.685645+0000","last_clean":"2024-03-20T08:46:29.685645+0000","last_became_active":"2024-03-20T08:45:51.647382+0000","last_became_peered":"2024-03-20T08:45:51.647382+0000","last_unstale":"2024-03-20T08:46:29.685645+0000","last_undegraded":"2024-03-20T08:46:29.685645+0000","last_fullsized":"2024-03-20T08:46:29.685645+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-20T08:45:50.616760+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:45:50.616760+0000","last_clean_scrub_stamp":"2024-03-20T08:45:50.616760+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T19:49:37.729239+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1c","version":"67'1","reported_seq":19,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-20T08:46:29.683504+0000","last_change":"2024-03-20T08:45:55.689327+0000","last_active":"2024-03-20T08:46:29.683504+0000","last_peered":"2024-03-20T08:46:29.683504+0000","last_clean":"2024-03-20T08:46:29.683504+0000","last_became_active":"2024-03-20T08:45:55.689061+0000","last_became_peered":"2024-03-20T08:45:55.689061+0000","last_unstale":"2024-03-20T08:46:29.683504+0000","last_undegraded":"2024-03-20T08:46:29.683504+0000","last_fullsized":"2024-03-20T08:46:29.683504+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-20T08:45:54.654511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:45:54.654511+0000","last_clean_scrub_stamp":"2024-03-20T08:45:54.654511+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T15:10:26.339680+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":370,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1f","version":"68'11","reported_seq":53,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-20T08:47:09.686638+0000","last_change":"2024-03-20T08:45:53.657892+0000","last_active":"2024-03-20T08:47:09.686638+0000","last_peered":"2024-03-20T08:47:09.686638+0000","last_clean":"2024-03-20T08:47:09.686638+0000","last_became_active":"2024-03-20T08:45:53.657554+0000","last_became_peered":"2024-03-20T08:45:53.657554+0000","last_unstale":"2024-03-20T08:47:09.686638+0000","last_undegraded":"2024-03-20T08:47:09.686638+0000","last_fullsized":"2024-03-20T08:47:09.686638+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-20T08:45:52.630650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:45:52.630650+0000","last_clean_scrub_stamp":"2024-03-20T08:45:52.630650+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T18:28:18.424683+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.19","version":"60'1","reported_seq":30,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-20T08:46:29.605687+0000","last_change":"2024-03-20T08:45:49.629593+0000","last_active":"2024-03-20T08:46:29.605687+0000","last_peered":"2024-03-20T08:46:29.605687+0000","last_clean":"2024-03-20T08:46:29.605687+0000","last_became_active":"2024-03-20T08:45:49.629341+0000","last_became_peered":"2024-03-20T08:45:49.629341+0000","last_unstale":"2024-03-20T08:46:29.605687+0000","last_undegraded":"2024-03-20T08:46:29.605687+0000","last_fullsized":"2024-03-20T08:46:29.605687+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-20T08:45:48.603868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:45:48.603868+0000","last_clean_scrub_stamp":"2024-03-20T08:45:48.603868+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T09:46:55.930165+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"67'10","reported_seq":40,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-20T08:46:29.610410+0000","last_change":"2024-03-20T08:45:51.648496+0000","last_active":"2024-03-20T08:46:29.610410+0000","last_peered":"2024-03-20T08:46:29.610410+0000","last_clean":"2024-03-20T08:46:29.610410+0000","last_became_active":"2024-03-20T08:45:51.648200+0000","last_became_peered":"2024-03-20T08:45:51.648200+0000","last_unstale":"2024-03-20T08:46:29.610410+0000","last_undegraded":"2024-03-20T08:46:29.610410+0000","last_fullsized":"2024-03-20T08:46:29.610410+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-20T08:45:50.616760+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:45:50.616760+0000","last_clean_scrub_stamp":"2024-03-20T08:45:50.616760+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T17:16:49.051084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":17,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-20T08:46:29.606693+0000","last_change":"2024-03-20T08:45:55.685786+0000","last_active":"2024-03-20T08:46:29.606693+0000","last_peered":"2024-03-20T08:46:29.606693+0000","last_clean":"2024-03-20T08:46:29.606693+0000","last_became_active":"2024-03-20T08:45:55.685448+0000","last_became_peered":"2024-03-20T08:45:55.685448+0000","last_unstale":"2024-03-20T08:46:29.606693+0000","last_undegraded":"2024-03-20T08:46:29.606693+0000","last_fullsized":"2024-03-20T08:46:29.606693+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-20T08:45:54.654511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:45:54.654511+0000","last_clean_scrub_stamp":"2024-03-20T08:45:54.654511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T15:28:04.468079+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":21,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-20T08:46:29.610694+0000","last_change":"2024-03-20T08:45:53.661895+0000","last_active":"2024-03-20T08:46:29.610694+0000","last_peered":"2024-03-20T08:46:29.610694+0000","last_clean":"2024-03-20T08:46:29.610694+0000","last_became_active":"2024-03-20T08:45:53.661544+0000","last_became_peered":"2024-03-20T08:45:53.661544+0000","last_unstale":"2024-03-20T08:46:29.610694+0000","last_undegraded":"2024-03-20T08:46:29.610694+0000","last_fullsized":"2024-03-20T08:46:29.610694+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-20T08:45:52.630650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:45:52.630650+0000","last_clean_scrub_stamp":"2024-03-20T08:45:52.630650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T10:42:03.766549+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":29,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-20T08:46:29.685737+0000","last_change":"2024-03-20T08:45:49.631944+0000","last_active":"2024-03-20T08:46:29.685737+0000","last_peered":"2024-03-20T08:46:29.685737+0000","last_clean":"2024-03-20T08:46:29.685737+0000","last_became_active":"2024-03-20T08:45:49.631611+0000","last_became_peered":"2024-03-20T08:45:49.631611+0000","last_unstale":"2024-03-20T08:46:29.685737+0000","last_undegraded":"2024-03-20T08:46:29.685737+0000","last_fullsized":"2024-03-20T08:46:29.685737+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-20T08:45:48.603868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:45:48.603868+0000","last_clean_scrub_stamp":"2024-03-20T08:45:48.603868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:37:25.584469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"67'11","reported_seq":44,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-20T08:46:29.684557+0000","last_change":"2024-03-20T08:45:51.646441+0000","last_active":"2024-03-20T08:46:29.684557+0000","last_peered":"2024-03-20T08:46:29.684557+0000","last_clean":"2024-03-20T08:46:29.684557+0000","last_became_active":"2024-03-20T08:45:51.646163+0000","last_became_peered":"2024-03-20T08:45:51.646163+0000","last_unstale":"2024-03-20T08:46:29.684557+0000","last_undegraded":"2024-03-20T08:46:29.684557+0000","last_fullsized":"2024-03-20T08:46:29.684557+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-20T08:45:50.616760+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:45:50.616760+0000","last_clean_scrub_stamp":"2024-03-20T08:45:50.616760+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T19:44:40.076880+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":17,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-20T08:46:29.683534+0000","last_change":"2024-03-20T08:45:55.689249+0000","last_active":"2024-03-20T08:46:29.683534+0000","last_peered":"2024-03-20T08:46:29.683534+0000","last_clean":"2024-03-20T08:46:29.683534+0000","last_became_active":"2024-03-20T08:45:55.688961+0000","last_became_peered":"2024-03-20T08:45:55.688961+0000","last_unstale":"2024-03-20T08:46:29.683534+0000","last_undegraded":"2024-03-20T08:46:29.683534+0000","last_fullsized":"2024-03-20T08:46:29.683534+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-20T08:45:54.654511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:45:54.654511+0000","last_clean_scrub_stamp":"2024-03-20T08:45:54.654511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T16:39:22.227095+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":21,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-20T08:46:29.683311+0000","last_change":"2024-03-20T08:45:53.666010+0000","last_active":"2024-03-20T08:46:29.683311+0000","last_peered":"2024-03-20T08:46:29.683311+0000","last_clean":"2024-03-20T08:46:29.683311+0000","last_became_active":"2024-03-20T08:45:53.665686+0000","last_became_peered":"2024-03-20T08:45:53.665686+0000","last_unstale":"2024-03-20T08:46:29.683311+0000","last_undegraded":"2024-03-20T08:46:29.683311+0000","last_fullsized":"2024-03-20T08:46:29.683311+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-20T08:45:52.630650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:45:52.630650+0000","last_clean_scrub_stamp":"2024-03-20T08:45:52.630650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:04:15.415014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":29,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-20T08:46:29.611333+0000","last_change":"2024-03-20T08:45:49.627794+0000","last_active":"2024-03-20T08:46:29.611333+0000","last_peered":"2024-03-20T08:46:29.611333+0000","last_clean":"2024-03-20T08:46:29.611333+0000","last_became_active":"2024-03-20T08:45:49.627500+0000","last_became_peered":"2024-03-20T08:45:49.627500+0000","last_unstale":"2024-03-20T08:46:29.611333+0000","last_undegraded":"2024-03-20T08:46:29.611333+0000","last_fullsized":"2024-03-20T08:46:29.611333+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-20T08:45:48.603868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:45:48.603868+0000","last_clean_scrub_stamp":"2024-03-20T08:45:48.603868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T17:11:32.430384+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"67'4","reported_seq":31,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-20T08:46:29.685907+0000","last_change":"2024-03-20T08:45:51.646793+0000","last_active":"2024-03-20T08:46:29.685907+0000","last_peered":"2024-03-20T08:46:29.685907+0000","last_clean":"2024-03-20T08:46:29.685907+0000","last_became_active":"2024-03-20T08:45:51.646478+0000","last_became_peered":"2024-03-20T08:45:51.646478+0000","last_unstale":"2024-03-20T08:46:29.685907+0000","last_undegraded":"2024-03-20T08:46:29.685907+0000","last_fullsized":"2024-03-20T08:46:29.685907+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-20T08:45:50.616760+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:45:50.616760+0000","last_clean_scrub_stamp":"2024-03-20T08:45:50.616760+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T16:26:49.832509+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":17,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-20T08:46:29.686665+0000","last_change":"2024-03-20T08:45:55.815899+0000","last_active":"2024-03-20T08:46:29.686665+0000","last_peered":"2024-03-20T08:46:29.686665+0000","last_clean":"2024-03-20T08:46:29.686665+0000","last_became_active":"2024-03-20T08:45:55.815317+0000","last_became_peered":"2024-03-20T08:45:55.815317+0000","last_unstale":"2024-03-20T08:46:29.686665+0000","last_undegraded":"2024-03-20T08:46:29.686665+0000","last_fullsized":"2024-03-20T08:46:29.686665+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-20T08:45:54.654511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:45:54.654511+0000","last_clean_scrub_stamp":"2024-03-20T08:45:54.654511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:13:59.756720+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":21,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-20T08:46:29.684413+0000","last_change":"2024-03-20T08:45:53.663516+0000","last_active":"2024-03-20T08:46:29.684413+0000","last_peered":"2024-03-20T08:46:29.684413+0000","last_clean":"2024-03-20T08:46:29.684413+0000","last_became_active":"2024-03-20T08:45:53.663221+0000","last_became_peered":"2024-03-20T08:45:53.663221+0000","last_unstale":"2024-03-20T08:46:29.684413+0000","last_undegraded":"2024-03-20T08:46:29.684413+0000","last_fullsized":"2024-03-20T08:46:29.684413+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-20T08:45:52.630650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:45:52.630650+0000","last_clean_scrub_stamp":"2024-03-20T08:45:52.630650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T16:46:27.073017+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.16","version":"60'1","reported_seq":37,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-20T08:46:29.607523+0000","last_change":"2024-03-20T08:45:49.633276+0000","last_active":"2024-03-20T08:46:29.607523+0000","last_peered":"2024-03-20T08:46:29.607523+0000","last_clean":"2024-03-20T08:46:29.607523+0000","last_became_active":"2024-03-20T08:45:49.632704+0000","last_became_peered":"2024-03-20T08:45:49.632704+0000","last_unstale":"2024-03-20T08:46:29.607523+0000","last_undegraded":"2024-03-20T08:46:29.607523+0000","last_fullsized":"2024-03-20T08:46:29.607523+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-20T08:45:48.603868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:45:48.603868+0000","last_clean_scrub_stamp":"2024-03-20T08:45:48.603868+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T10:39:25.298178+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":911,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.11","version":"67'11","reported_seq":44,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-20T08:46:29.685082+0000","last_change":"2024-03-20T08:45:51.650889+0000","last_active":"2024-03-20T08:46:29.685082+0000","last_peered":"2024-03-20T08:46:29.685082+0000","last_clean":"2024-03-20T08:46:29.685082+0000","last_became_active":"2024-03-20T08:45:51.650596+0000","last_became_peered":"2024-03-20T08:45:51.650596+0000","last_unstale":"2024-03-20T08:46:29.685082+0000","last_undegraded":"2024-03-20T08:46:29.685082+0000","last_fullsized":"2024-03-20T08:46:29.685082+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-20T08:45:50.616760+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:45:50.616760+0000","last_clean_scrub_stamp":"2024-03-20T08:45:50.616760+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T11:22:00.070168+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":17,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-20T08:46:29.609700+0000","last_change":"2024-03-20T08:45:55.687298+0000","last_active":"2024-03-20T08:46:29.609700+0000","last_peered":"2024-03-20T08:46:29.609700+0000","last_clean":"2024-03-20T08:46:29.609700+0000","last_became_active":"2024-03-20T08:45:55.686910+0000","last_became_peered":"2024-03-20T08:45:55.686910+0000","last_unstale":"2024-03-20T08:46:29.609700+0000","last_undegraded":"2024-03-20T08:46:29.609700+0000","last_fullsized":"2024-03-20T08:46:29.609700+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-20T08:45:54.654511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:45:54.654511+0000","last_clean_scrub_stamp":"2024-03-20T08:45:54.654511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T11:31:36.817467+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":21,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-20T08:46:29.686938+0000","last_change":"2024-03-20T08:45:53.655655+0000","last_active":"2024-03-20T08:46:29.686938+0000","last_peered":"2024-03-20T08:46:29.686938+0000","last_clean":"2024-03-20T08:46:29.686938+0000","last_became_active":"2024-03-20T08:45:53.655269+0000","last_became_peered":"2024-03-20T08:45:53.655269+0000","last_unstale":"2024-03-20T08:46:29.686938+0000","last_undegraded":"2024-03-20T08:46:29.686938+0000","last_fullsized":"2024-03-20T08:46:29.686938+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-20T08:45:52.630650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:45:52.630650+0000","last_clean_scrub_stamp":"2024-03-20T08:45:52.630650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:07:06.161785+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.15","version":"60'1","reported_seq":37,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-20T08:46:29.684666+0000","last_change":"2024-03-20T08:45:49.629961+0000","last_active":"2024-03-20T08:46:29.684666+0000","last_peered":"2024-03-20T08:46:29.684666+0000","last_clean":"2024-03-20T08:46:29.684666+0000","last_became_active":"2024-03-20T08:45:49.629491+0000","last_became_peered":"2024-03-20T08:45:49.629491+0000","last_unstale":"2024-03-20T08:46:29.684666+0000","last_undegraded":"2024-03-20T08:46:29.684666+0000","last_fullsized":"2024-03-20T08:46:29.684666+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-20T08:45:48.603868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:45:48.603868+0000","last_clean_scrub_stamp":"2024-03-20T08:45:48.603868+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T17:07:22.223272+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":436,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.12","version":"67'9","reported_seq":41,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-20T08:46:29.606473+0000","last_change":"2024-03-20T08:45:51.642899+0000","last_active":"2024-03-20T08:46:29.606473+0000","last_peered":"2024-03-20T08:46:29.606473+0000","last_clean":"2024-03-20T08:46:29.606473+0000","last_became_active":"2024-03-20T08:45:51.642610+0000","last_became_peered":"2024-03-20T08:45:51.642610+0000","last_unstale":"2024-03-20T08:46:29.606473+0000","last_undegraded":"2024-03-20T08:46:29.606473+0000","last_fullsized":"2024-03-20T08:46:29.606473+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-20T08:45:50.616760+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:45:50.616760+0000","last_clean_scrub_stamp":"2024-03-20T08:45:50.616760+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T12:24:22.517141+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":17,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-20T08:46:29.684780+0000","last_change":"2024-03-20T08:45:55.692907+0000","last_active":"2024-03-20T08:46:29.684780+0000","last_peered":"2024-03-20T08:46:29.684780+0000","last_clean":"2024-03-20T08:46:29.684780+0000","last_became_active":"2024-03-20T08:45:55.691558+0000","last_became_peered":"2024-03-20T08:45:55.691558+0000","last_unstale":"2024-03-20T08:46:29.684780+0000","last_undegraded":"2024-03-20T08:46:29.684780+0000","last_fullsized":"2024-03-20T08:46:29.684780+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-20T08:45:54.654511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:45:54.654511+0000","last_clean_scrub_stamp":"2024-03-20T08:45:54.654511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T19:33:31.509549+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":21,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-20T08:46:29.606246+0000","last_change":"2024-03-20T08:45:53.657596+0000","last_active":"2024-03-20T08:46:29.606246+0000","last_peered":"2024-03-20T08:46:29.606246+0000","last_clean":"2024-03-20T08:46:29.606246+0000","last_became_active":"2024-03-20T08:45:53.656954+0000","last_became_peered":"2024-03-20T08:45:53.656954+0000","last_unstale":"2024-03-20T08:46:29.606246+0000","last_undegraded":"2024-03-20T08:46:29.606246+0000","last_fullsized":"2024-03-20T08:46:29.606246+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-20T08:45:52.630650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:45:52.630650+0000","last_clean_scrub_stamp":"2024-03-20T08:45:52.630650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T10:31:30.378735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":29,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-20T08:46:29.659254+0000","last_change":"2024-03-20T08:45:49.629013+0000","last_active":"2024-03-20T08:46:29.659254+0000","last_peered":"2024-03-20T08:46:29.659254+0000","last_clean":"2024-03-20T08:46:29.659254+0000","last_became_active":"2024-03-20T08:45:49.628287+0000","last_became_peered":"2024-03-20T08:45:49.628287+0000","last_unstale":"2024-03-20T08:46:29.659254+0000","last_undegraded":"2024-03-20T08:46:29.659254+0000","last_fullsized":"2024-03-20T08:46:29.659254+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-20T08:45:48.603868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:45:48.603868+0000","last_clean_scrub_stamp":"2024-03-20T08:45:48.603868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T17:14:24.813516+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"67'11","reported_seq":44,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-20T08:46:29.659050+0000","last_change":"2024-03-20T08:45:51.652783+0000","last_active":"2024-03-20T08:46:29.659050+0000","last_peered":"2024-03-20T08:46:29.659050+0000","last_clean":"2024-03-20T08:46:29.659050+0000","last_became_active":"2024-03-20T08:45:51.652485+0000","last_became_peered":"2024-03-20T08:45:51.652485+0000","last_unstale":"2024-03-20T08:46:29.659050+0000","last_undegraded":"2024-03-20T08:46:29.659050+0000","last_fullsized":"2024-03-20T08:46:29.659050+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-20T08:45:50.616760+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:45:50.616760+0000","last_clean_scrub_stamp":"2024-03-20T08:45:50.616760+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:22:17.120553+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":17,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-20T08:46:29.610029+0000","last_change":"2024-03-20T08:45:55.693809+0000","last_active":"2024-03-20T08:46:29.610029+0000","last_peered":"2024-03-20T08:46:29.610029+0000","last_clean":"2024-03-20T08:46:29.610029+0000","last_became_active":"2024-03-20T08:45:55.693482+0000","last_became_peered":"2024-03-20T08:45:55.693482+0000","last_unstale":"2024-03-20T08:46:29.610029+0000","last_undegraded":"2024-03-20T08:46:29.610029+0000","last_fullsized":"2024-03-20T08:46:29.610029+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-20T08:45:54.654511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:45:54.654511+0000","last_clean_scrub_stamp":"2024-03-20T08:45:54.654511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T15:59:51.338868+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.15","version":"68'11","reported_seq":50,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-20T08:47:09.686639+0000","last_change":"2024-03-20T08:45:53.663741+0000","last_active":"2024-03-20T08:47:09.686639+0000","last_peered":"2024-03-20T08:47:09.686639+0000","last_clean":"2024-03-20T08:47:09.686639+0000","last_became_active":"2024-03-20T08:45:53.663452+0000","last_became_peered":"2024-03-20T08:45:53.663452+0000","last_unstale":"2024-03-20T08:47:09.686639+0000","last_undegraded":"2024-03-20T08:47:09.686639+0000","last_fullsized":"2024-03-20T08:47:09.686639+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-20T08:45:52.630650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:45:52.630650+0000","last_clean_scrub_stamp":"2024-03-20T08:45:52.630650+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T16:42:16.218775+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":29,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-20T08:46:29.683850+0000","last_change":"2024-03-20T08:45:49.631507+0000","last_active":"2024-03-20T08:46:29.683850+0000","last_peered":"2024-03-20T08:46:29.683850+0000","last_clean":"2024-03-20T08:46:29.683850+0000","last_became_active":"2024-03-20T08:45:49.631193+0000","last_became_peered":"2024-03-20T08:45:49.631193+0000","last_unstale":"2024-03-20T08:46:29.683850+0000","last_undegraded":"2024-03-20T08:46:29.683850+0000","last_fullsized":"2024-03-20T08:46:29.683850+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-20T08:45:48.603868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:45:48.603868+0000","last_clean_scrub_stamp":"2024-03-20T08:45:48.603868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T18:58:24.656178+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"67'10","reported_seq":40,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-20T08:46:29.687033+0000","last_change":"2024-03-20T08:45:51.649967+0000","last_active":"2024-03-20T08:46:29.687033+0000","last_peered":"2024-03-20T08:46:29.687033+0000","last_clean":"2024-03-20T08:46:29.687033+0000","last_became_active":"2024-03-20T08:45:51.649657+0000","last_became_peered":"2024-03-20T08:45:51.649657+0000","last_unstale":"2024-03-20T08:46:29.687033+0000","last_undegraded":"2024-03-20T08:46:29.687033+0000","last_fullsized":"2024-03-20T08:46:29.687033+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-20T08:45:50.616760+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:45:50.616760+0000","last_clean_scrub_stamp":"2024-03-20T08:45:50.616760+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T19:34:55.533460+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":17,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-20T08:46:29.659612+0000","last_change":"2024-03-20T08:45:55.687183+0000","last_active":"2024-03-20T08:46:29.659612+0000","last_peered":"2024-03-20T08:46:29.659612+0000","last_clean":"2024-03-20T08:46:29.659612+0000","last_became_active":"2024-03-20T08:45:55.686757+0000","last_became_peered":"2024-03-20T08:45:55.686757+0000","last_unstale":"2024-03-20T08:46:29.659612+0000","last_undegraded":"2024-03-20T08:46:29.659612+0000","last_fullsized":"2024-03-20T08:46:29.659612+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-20T08:45:54.654511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:45:54.654511+0000","last_clean_scrub_stamp":"2024-03-20T08:45:54.654511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:14:14.232613+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.14","version":"68'11","reported_seq":50,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-20T08:47:09.686782+0000","last_change":"2024-03-20T08:45:53.665034+0000","last_active":"2024-03-20T08:47:09.686782+0000","last_peered":"2024-03-20T08:47:09.686782+0000","last_clean":"2024-03-20T08:47:09.686782+0000","last_became_active":"2024-03-20T08:45:53.664661+0000","last_became_peered":"2024-03-20T08:45:53.664661+0000","last_unstale":"2024-03-20T08:47:09.686782+0000","last_undegraded":"2024-03-20T08:47:09.686782+0000","last_fullsized":"2024-03-20T08:47:09.686782+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-20T08:45:52.630650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:45:52.630650+0000","last_clean_scrub_stamp":"2024-03-20T08:45:52.630650+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T19:43:34.419422+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":29,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-20T08:46:29.610622+0000","last_change":"2024-03-20T08:45:49.628884+0000","last_active":"2024-03-20T08:46:29.610622+0000","last_peered":"2024-03-20T08:46:29.610622+0000","last_clean":"2024-03-20T08:46:29.610622+0000","last_became_active":"2024-03-20T08:45:49.628553+0000","last_became_peered":"2024-03-20T08:45:49.628553+0000","last_unstale":"2024-03-20T08:46:29.610622+0000","last_undegraded":"2024-03-20T08:46:29.610622+0000","last_fullsized":"2024-03-20T08:46:29.610622+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-20T08:45:48.603868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:45:48.603868+0000","last_clean_scrub_stamp":"2024-03-20T08:45:48.603868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T12:59:19.958241+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"67'9","reported_seq":41,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-20T08:46:29.607591+0000","last_change":"2024-03-20T08:45:51.645538+0000","last_active":"2024-03-20T08:46:29.607591+0000","last_peered":"2024-03-20T08:46:29.607591+0000","last_clean":"2024-03-20T08:46:29.607591+0000","last_became_active":"2024-03-20T08:45:51.644458+0000","last_became_peered":"2024-03-20T08:45:51.644458+0000","last_unstale":"2024-03-20T08:46:29.607591+0000","last_undegraded":"2024-03-20T08:46:29.607591+0000","last_fullsized":"2024-03-20T08:46:29.607591+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-20T08:45:50.616760+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:45:50.616760+0000","last_clean_scrub_stamp":"2024-03-20T08:45:50.616760+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T18:21:22.224425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":17,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-20T08:46:29.684969+0000","last_change":"2024-03-20T08:45:55.689296+0000","last_active":"2024-03-20T08:46:29.684969+0000","last_peered":"2024-03-20T08:46:29.684969+0000","last_clean":"2024-03-20T08:46:29.684969+0000","last_became_active":"2024-03-20T08:45:55.688994+0000","last_became_peered":"2024-03-20T08:45:55.688994+0000","last_unstale":"2024-03-20T08:46:29.684969+0000","last_undegraded":"2024-03-20T08:46:29.684969+0000","last_fullsized":"2024-03-20T08:46:29.684969+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-20T08:45:54.654511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:45:54.654511+0000","last_clean_scrub_stamp":"2024-03-20T08:45:54.654511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T11:06:35.965720+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":21,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-20T08:46:29.687582+0000","last_change":"2024-03-20T08:45:53.660837+0000","last_active":"2024-03-20T08:46:29.687582+0000","last_peered":"2024-03-20T08:46:29.687582+0000","last_clean":"2024-03-20T08:46:29.687582+0000","last_became_active":"2024-03-20T08:45:53.660512+0000","last_became_peered":"2024-03-20T08:45:53.660512+0000","last_unstale":"2024-03-20T08:46:29.687582+0000","last_undegraded":"2024-03-20T08:46:29.687582+0000","last_fullsized":"2024-03-20T08:46:29.687582+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-20T08:45:52.630650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:45:52.630650+0000","last_clean_scrub_stamp":"2024-03-20T08:45:52.630650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T17:46:25.157304+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":29,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-20T08:46:29.684066+0000","last_change":"2024-03-20T08:45:49.635863+0000","last_active":"2024-03-20T08:46:29.684066+0000","last_peered":"2024-03-20T08:46:29.684066+0000","last_clean":"2024-03-20T08:46:29.684066+0000","last_became_active":"2024-03-20T08:45:49.635570+0000","last_became_peered":"2024-03-20T08:45:49.635570+0000","last_unstale":"2024-03-20T08:46:29.684066+0000","last_undegraded":"2024-03-20T08:46:29.684066+0000","last_fullsized":"2024-03-20T08:46:29.684066+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-20T08:45:48.603868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:45:48.603868+0000","last_clean_scrub_stamp":"2024-03-20T08:45:48.603868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T09:38:45.343126+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"67'9","reported_seq":41,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-20T08:46:29.610525+0000","last_change":"2024-03-20T08:45:51.648332+0000","last_active":"2024-03-20T08:46:29.610525+0000","last_peered":"2024-03-20T08:46:29.610525+0000","last_clean":"2024-03-20T08:46:29.610525+0000","last_became_active":"2024-03-20T08:45:51.648062+0000","last_became_peered":"2024-03-20T08:45:51.648062+0000","last_unstale":"2024-03-20T08:46:29.610525+0000","last_undegraded":"2024-03-20T08:46:29.610525+0000","last_fullsized":"2024-03-20T08:46:29.610525+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-20T08:45:50.616760+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:45:50.616760+0000","last_clean_scrub_stamp":"2024-03-20T08:45:50.616760+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T16:39:51.402299+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":17,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-20T08:46:29.684579+0000","last_change":"2024-03-20T08:45:55.816865+0000","last_active":"2024-03-20T08:46:29.684579+0000","last_peered":"2024-03-20T08:46:29.684579+0000","last_clean":"2024-03-20T08:46:29.684579+0000","last_became_active":"2024-03-20T08:45:55.816554+0000","last_became_peered":"2024-03-20T08:45:55.816554+0000","last_unstale":"2024-03-20T08:46:29.684579+0000","last_undegraded":"2024-03-20T08:46:29.684579+0000","last_fullsized":"2024-03-20T08:46:29.684579+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-20T08:45:54.654511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:45:54.654511+0000","last_clean_scrub_stamp":"2024-03-20T08:45:54.654511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:27:53.618904+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":21,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-20T08:46:29.607637+0000","last_change":"2024-03-20T08:45:53.660167+0000","last_active":"2024-03-20T08:46:29.607637+0000","last_peered":"2024-03-20T08:46:29.607637+0000","last_clean":"2024-03-20T08:46:29.607637+0000","last_became_active":"2024-03-20T08:45:53.659874+0000","last_became_peered":"2024-03-20T08:45:53.659874+0000","last_unstale":"2024-03-20T08:46:29.607637+0000","last_undegraded":"2024-03-20T08:46:29.607637+0000","last_fullsized":"2024-03-20T08:46:29.607637+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-20T08:45:52.630650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:45:52.630650+0000","last_clean_scrub_stamp":"2024-03-20T08:45:52.630650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T10:17:48.470760+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":29,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-20T08:46:29.683184+0000","last_change":"2024-03-20T08:45:49.627698+0000","last_active":"2024-03-20T08:46:29.683184+0000","last_peered":"2024-03-20T08:46:29.683184+0000","last_clean":"2024-03-20T08:46:29.683184+0000","last_became_active":"2024-03-20T08:45:49.627468+0000","last_became_peered":"2024-03-20T08:45:49.627468+0000","last_unstale":"2024-03-20T08:46:29.683184+0000","last_undegraded":"2024-03-20T08:46:29.683184+0000","last_fullsized":"2024-03-20T08:46:29.683184+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-20T08:45:48.603868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:45:48.603868+0000","last_clean_scrub_stamp":"2024-03-20T08:45:48.603868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T18:00:26.933257+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"67'6","reported_seq":34,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-20T08:46:29.687463+0000","last_change":"2024-03-20T08:45:51.650932+0000","last_active":"2024-03-20T08:46:29.687463+0000","last_peered":"2024-03-20T08:46:29.687463+0000","last_clean":"2024-03-20T08:46:29.687463+0000","last_became_active":"2024-03-20T08:45:51.650635+0000","last_became_peered":"2024-03-20T08:45:51.650635+0000","last_unstale":"2024-03-20T08:46:29.687463+0000","last_undegraded":"2024-03-20T08:46:29.687463+0000","last_fullsized":"2024-03-20T08:46:29.687463+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-20T08:45:50.616760+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:45:50.616760+0000","last_clean_scrub_stamp":"2024-03-20T08:45:50.616760+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T09:20:24.448441+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":9,"num_read_kb":6,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":17,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-20T08:46:29.607106+0000","last_change":"2024-03-20T08:45:55.814872+0000","last_active":"2024-03-20T08:46:29.607106+0000","last_peered":"2024-03-20T08:46:29.607106+0000","last_clean":"2024-03-20T08:46:29.607106+0000","last_became_active":"2024-03-20T08:45:55.814481+0000","last_became_peered":"2024-03-20T08:45:55.814481+0000","last_unstale":"2024-03-20T08:46:29.607106+0000","last_undegraded":"2024-03-20T08:46:29.607106+0000","last_fullsized":"2024-03-20T08:46:29.607106+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-20T08:45:54.654511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:45:54.654511+0000","last_clean_scrub_stamp":"2024-03-20T08:45:54.654511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T12:49:29.493130+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"60'2","reported_seq":45,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-20T08:46:29.683555+0000","last_change":"2024-03-20T08:45:49.634369+0000","last_active":"2024-03-20T08:46:29.683555+0000","last_peered":"2024-03-20T08:46:29.683555+0000","last_clean":"2024-03-20T08:46:29.683555+0000","last_became_active":"2024-03-20T08:45:49.633882+0000","last_became_peered":"2024-03-20T08:45:49.633882+0000","last_unstale":"2024-03-20T08:46:29.683555+0000","last_undegraded":"2024-03-20T08:46:29.683555+0000","last_fullsized":"2024-03-20T08:46:29.683555+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-20T08:45:48.603868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:45:48.603868+0000","last_clean_scrub_stamp":"2024-03-20T08:45:48.603868+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T11:22:20.528408+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":14,"num_read_kb":14,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"67'15","reported_seq":50,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-20T08:46:29.606618+0000","last_change":"2024-03-20T08:45:51.641571+0000","last_active":"2024-03-20T08:46:29.606618+0000","last_peered":"2024-03-20T08:46:29.606618+0000","last_clean":"2024-03-20T08:46:29.606618+0000","last_became_active":"2024-03-20T08:45:51.641144+0000","last_became_peered":"2024-03-20T08:45:51.641144+0000","last_unstale":"2024-03-20T08:46:29.606618+0000","last_undegraded":"2024-03-20T08:46:29.606618+0000","last_fullsized":"2024-03-20T08:46:29.606618+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-20T08:45:50.616760+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:45:50.616760+0000","last_clean_scrub_stamp":"2024-03-20T08:45:50.616760+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T16:35:21.951940+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"68'11","reported_seq":50,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-20T08:47:09.686639+0000","last_change":"2024-03-20T08:45:53.665167+0000","last_active":"2024-03-20T08:47:09.686639+0000","last_peered":"2024-03-20T08:47:09.686639+0000","last_clean":"2024-03-20T08:47:09.686639+0000","last_became_active":"2024-03-20T08:45:53.664813+0000","last_became_peered":"2024-03-20T08:45:53.664813+0000","last_unstale":"2024-03-20T08:47:09.686639+0000","last_undegraded":"2024-03-20T08:47:09.686639+0000","last_fullsized":"2024-03-20T08:47:09.686639+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-20T08:45:52.630650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:45:52.630650+0000","last_clean_scrub_stamp":"2024-03-20T08:45:52.630650+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T11:17:18.224664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":17,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-20T08:46:29.686756+0000","last_change":"2024-03-20T08:45:55.693636+0000","last_active":"2024-03-20T08:46:29.686756+0000","last_peered":"2024-03-20T08:46:29.686756+0000","last_clean":"2024-03-20T08:46:29.686756+0000","last_became_active":"2024-03-20T08:45:55.693321+0000","last_became_peered":"2024-03-20T08:45:55.693321+0000","last_unstale":"2024-03-20T08:46:29.686756+0000","last_undegraded":"2024-03-20T08:46:29.686756+0000","last_fullsized":"2024-03-20T08:46:29.686756+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-20T08:45:54.654511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:45:54.654511+0000","last_clean_scrub_stamp":"2024-03-20T08:45:54.654511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T18:35:56.615151+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":29,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-20T08:46:29.683788+0000","last_change":"2024-03-20T08:45:49.631139+0000","last_active":"2024-03-20T08:46:29.683788+0000","last_peered":"2024-03-20T08:46:29.683788+0000","last_clean":"2024-03-20T08:46:29.683788+0000","last_became_active":"2024-03-20T08:45:49.630850+0000","last_became_peered":"2024-03-20T08:45:49.630850+0000","last_unstale":"2024-03-20T08:46:29.683788+0000","last_undegraded":"2024-03-20T08:46:29.683788+0000","last_fullsized":"2024-03-20T08:46:29.683788+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-20T08:45:48.603868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:45:48.603868+0000","last_clean_scrub_stamp":"2024-03-20T08:45:48.603868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T18:23:48.781961+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"67'12","reported_seq":48,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-20T08:46:29.658895+0000","last_change":"2024-03-20T08:45:51.648425+0000","last_active":"2024-03-20T08:46:29.658895+0000","last_peered":"2024-03-20T08:46:29.658895+0000","last_clean":"2024-03-20T08:46:29.658895+0000","last_became_active":"2024-03-20T08:45:51.648094+0000","last_became_peered":"2024-03-20T08:45:51.648094+0000","last_unstale":"2024-03-20T08:46:29.658895+0000","last_undegraded":"2024-03-20T08:46:29.658895+0000","last_fullsized":"2024-03-20T08:46:29.658895+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-20T08:45:50.616760+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:45:50.616760+0000","last_clean_scrub_stamp":"2024-03-20T08:45:50.616760+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T20:24:59.105925+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":21,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-20T08:46:29.684269+0000","last_change":"2024-03-20T08:45:53.667088+0000","last_active":"2024-03-20T08:46:29.684269+0000","last_peered":"2024-03-20T08:46:29.684269+0000","last_clean":"2024-03-20T08:46:29.684269+0000","last_became_active":"2024-03-20T08:45:53.666769+0000","last_became_peered":"2024-03-20T08:45:53.666769+0000","last_unstale":"2024-03-20T08:46:29.684269+0000","last_undegraded":"2024-03-20T08:46:29.684269+0000","last_fullsized":"2024-03-20T08:46:29.684269+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-20T08:45:52.630650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:45:52.630650+0000","last_clean_scrub_stamp":"2024-03-20T08:45:52.630650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:12:02.183265+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":17,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-20T08:46:29.683649+0000","last_change":"2024-03-20T08:45:55.693463+0000","last_active":"2024-03-20T08:46:29.683649+0000","last_peered":"2024-03-20T08:46:29.683649+0000","last_clean":"2024-03-20T08:46:29.683649+0000","last_became_active":"2024-03-20T08:45:55.693220+0000","last_became_peered":"2024-03-20T08:45:55.693220+0000","last_unstale":"2024-03-20T08:46:29.683649+0000","last_undegraded":"2024-03-20T08:46:29.683649+0000","last_fullsized":"2024-03-20T08:46:29.683649+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-20T08:45:54.654511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:45:54.654511+0000","last_clean_scrub_stamp":"2024-03-20T08:45:54.654511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:15:44.654763+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":21,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-20T08:46:29.684225+0000","last_change":"2024-03-20T08:45:53.667191+0000","last_active":"2024-03-20T08:46:29.684225+0000","last_peered":"2024-03-20T08:46:29.684225+0000","last_clean":"2024-03-20T08:46:29.684225+0000","last_became_active":"2024-03-20T08:45:53.666883+0000","last_became_peered":"2024-03-20T08:45:53.666883+0000","last_unstale":"2024-03-20T08:46:29.684225+0000","last_undegraded":"2024-03-20T08:46:29.684225+0000","last_fullsized":"2024-03-20T08:46:29.684225+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-20T08:45:52.630650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:45:52.630650+0000","last_clean_scrub_stamp":"2024-03-20T08:45:52.630650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T15:06:50.829308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":29,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-20T08:46:29.683217+0000","last_change":"2024-03-20T08:45:49.635051+0000","last_active":"2024-03-20T08:46:29.683217+0000","last_peered":"2024-03-20T08:46:29.683217+0000","last_clean":"2024-03-20T08:46:29.683217+0000","last_became_active":"2024-03-20T08:45:49.634085+0000","last_became_peered":"2024-03-20T08:45:49.634085+0000","last_unstale":"2024-03-20T08:46:29.683217+0000","last_undegraded":"2024-03-20T08:46:29.683217+0000","last_fullsized":"2024-03-20T08:46:29.683217+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-20T08:45:48.603868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:45:48.603868+0000","last_clean_scrub_stamp":"2024-03-20T08:45:48.603868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T16:36:22.957495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"67'19","reported_seq":56,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-20T08:46:29.684043+0000","last_change":"2024-03-20T08:45:51.646441+0000","last_active":"2024-03-20T08:46:29.684043+0000","last_peered":"2024-03-20T08:46:29.684043+0000","last_clean":"2024-03-20T08:46:29.684043+0000","last_became_active":"2024-03-20T08:45:51.646126+0000","last_became_peered":"2024-03-20T08:45:51.646126+0000","last_unstale":"2024-03-20T08:46:29.684043+0000","last_undegraded":"2024-03-20T08:46:29.684043+0000","last_fullsized":"2024-03-20T08:46:29.684043+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-20T08:45:50.616760+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:45:50.616760+0000","last_clean_scrub_stamp":"2024-03-20T08:45:50.616760+0000","objects_scrubbed":0,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T09:59:33.305144+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":32,"num_read_kb":21,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":17,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-20T08:46:29.610157+0000","last_change":"2024-03-20T08:45:55.687980+0000","last_active":"2024-03-20T08:46:29.610157+0000","last_peered":"2024-03-20T08:46:29.610157+0000","last_clean":"2024-03-20T08:46:29.610157+0000","last_became_active":"2024-03-20T08:45:55.687230+0000","last_became_peered":"2024-03-20T08:45:55.687230+0000","last_unstale":"2024-03-20T08:46:29.610157+0000","last_undegraded":"2024-03-20T08:46:29.610157+0000","last_fullsized":"2024-03-20T08:46:29.610157+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-20T08:45:54.654511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:45:54.654511+0000","last_clean_scrub_stamp":"2024-03-20T08:45:54.654511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T09:33:04.045483+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":21,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-20T08:46:29.684326+0000","last_change":"2024-03-20T08:45:53.665316+0000","last_active":"2024-03-20T08:46:29.684326+0000","last_peered":"2024-03-20T08:46:29.684326+0000","last_clean":"2024-03-20T08:46:29.684326+0000","last_became_active":"2024-03-20T08:45:53.664079+0000","last_became_peered":"2024-03-20T08:45:53.664079+0000","last_unstale":"2024-03-20T08:46:29.684326+0000","last_undegraded":"2024-03-20T08:46:29.684326+0000","last_fullsized":"2024-03-20T08:46:29.684326+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-20T08:45:52.630650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:45:52.630650+0000","last_clean_scrub_stamp":"2024-03-20T08:45:52.630650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:28:50.692891+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":29,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-20T08:46:29.607263+0000","last_change":"2024-03-20T08:45:49.630947+0000","last_active":"2024-03-20T08:46:29.607263+0000","last_peered":"2024-03-20T08:46:29.607263+0000","last_clean":"2024-03-20T08:46:29.607263+0000","last_became_active":"2024-03-20T08:45:49.630606+0000","last_became_peered":"2024-03-20T08:45:49.630606+0000","last_unstale":"2024-03-20T08:46:29.607263+0000","last_undegraded":"2024-03-20T08:46:29.607263+0000","last_fullsized":"2024-03-20T08:46:29.607263+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-20T08:45:48.603868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:45:48.603868+0000","last_clean_scrub_stamp":"2024-03-20T08:45:48.603868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T17:21:13.198286+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"67'9","reported_seq":41,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-20T08:46:29.611339+0000","last_change":"2024-03-20T08:45:51.643356+0000","last_active":"2024-03-20T08:46:29.611339+0000","last_peered":"2024-03-20T08:46:29.611339+0000","last_clean":"2024-03-20T08:46:29.611339+0000","last_became_active":"2024-03-20T08:45:51.643069+0000","last_became_peered":"2024-03-20T08:45:51.643069+0000","last_unstale":"2024-03-20T08:46:29.611339+0000","last_undegraded":"2024-03-20T08:46:29.611339+0000","last_fullsized":"2024-03-20T08:46:29.611339+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-20T08:45:50.616760+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:45:50.616760+0000","last_clean_scrub_stamp":"2024-03-20T08:45:50.616760+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T19:30:50.784255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":17,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-20T08:46:29.684553+0000","last_change":"2024-03-20T08:45:55.694077+0000","last_active":"2024-03-20T08:46:29.684553+0000","last_peered":"2024-03-20T08:46:29.684553+0000","last_clean":"2024-03-20T08:46:29.684553+0000","last_became_active":"2024-03-20T08:45:55.693647+0000","last_became_peered":"2024-03-20T08:45:55.693647+0000","last_unstale":"2024-03-20T08:46:29.684553+0000","last_undegraded":"2024-03-20T08:46:29.684553+0000","last_fullsized":"2024-03-20T08:46:29.684553+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-20T08:45:54.654511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:45:54.654511+0000","last_clean_scrub_stamp":"2024-03-20T08:45:54.654511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T16:51:24.976022+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"64'2","reported_seq":38,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-20T08:46:29.606814+0000","last_change":"2024-03-20T08:45:53.655794+0000","last_active":"2024-03-20T08:46:29.606814+0000","last_peered":"2024-03-20T08:46:29.606814+0000","last_clean":"2024-03-20T08:46:29.606814+0000","last_became_active":"2024-03-20T08:45:49.628760+0000","last_became_peered":"2024-03-20T08:45:49.628760+0000","last_unstale":"2024-03-20T08:46:29.606814+0000","last_undegraded":"2024-03-20T08:46:29.606814+0000","last_fullsized":"2024-03-20T08:46:29.606814+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-20T08:45:48.603868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:45:48.603868+0000","last_clean_scrub_stamp":"2024-03-20T08:45:48.603868+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T15:41:27.543966+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0015906500000000001,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":29,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-20T08:46:29.658330+0000","last_change":"2024-03-20T08:45:49.631750+0000","last_active":"2024-03-20T08:46:29.658330+0000","last_peered":"2024-03-20T08:46:29.658330+0000","last_clean":"2024-03-20T08:46:29.658330+0000","last_became_active":"2024-03-20T08:45:49.630383+0000","last_became_peered":"2024-03-20T08:45:49.630383+0000","last_unstale":"2024-03-20T08:46:29.658330+0000","last_undegraded":"2024-03-20T08:46:29.658330+0000","last_fullsized":"2024-03-20T08:46:29.658330+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-20T08:45:48.603868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:45:48.603868+0000","last_clean_scrub_stamp":"2024-03-20T08:45:48.603868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T17:13:00.588163+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.4","version":"68'30","reported_seq":94,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-20T08:47:09.686651+0000","last_change":"2024-03-20T08:45:51.646326+0000","last_active":"2024-03-20T08:47:09.686651+0000","last_peered":"2024-03-20T08:47:09.686651+0000","last_clean":"2024-03-20T08:47:09.686651+0000","last_became_active":"2024-03-20T08:45:51.646030+0000","last_became_peered":"2024-03-20T08:45:51.646030+0000","last_unstale":"2024-03-20T08:47:09.686651+0000","last_undegraded":"2024-03-20T08:47:09.686651+0000","last_fullsized":"2024-03-20T08:47:09.686651+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-20T08:45:50.616760+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:45:50.616760+0000","last_clean_scrub_stamp":"2024-03-20T08:45:50.616760+0000","objects_scrubbed":0,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T15:45:40.271612+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":51,"num_read_kb":36,"num_write":26,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":21,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-20T08:46:29.611015+0000","last_change":"2024-03-20T08:45:53.655027+0000","last_active":"2024-03-20T08:46:29.611015+0000","last_peered":"2024-03-20T08:46:29.611015+0000","last_clean":"2024-03-20T08:46:29.611015+0000","last_became_active":"2024-03-20T08:45:53.654604+0000","last_became_peered":"2024-03-20T08:45:53.654604+0000","last_unstale":"2024-03-20T08:46:29.611015+0000","last_undegraded":"2024-03-20T08:46:29.611015+0000","last_fullsized":"2024-03-20T08:46:29.611015+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-20T08:45:52.630650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:45:52.630650+0000","last_clean_scrub_stamp":"2024-03-20T08:45:52.630650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T15:10:01.286030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":17,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-20T08:46:29.684821+0000","last_change":"2024-03-20T08:45:55.817347+0000","last_active":"2024-03-20T08:46:29.684821+0000","last_peered":"2024-03-20T08:46:29.684821+0000","last_clean":"2024-03-20T08:46:29.684821+0000","last_became_active":"2024-03-20T08:45:55.816689+0000","last_became_peered":"2024-03-20T08:45:55.816689+0000","last_unstale":"2024-03-20T08:46:29.684821+0000","last_undegraded":"2024-03-20T08:46:29.684821+0000","last_fullsized":"2024-03-20T08:46:29.684821+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-20T08:45:54.654511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:45:54.654511+0000","last_clean_scrub_stamp":"2024-03-20T08:45:54.654511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T16:03:23.078938+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":29,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-20T08:46:29.605660+0000","last_change":"2024-03-20T08:45:49.629091+0000","last_active":"2024-03-20T08:46:29.605660+0000","last_peered":"2024-03-20T08:46:29.605660+0000","last_clean":"2024-03-20T08:46:29.605660+0000","last_became_active":"2024-03-20T08:45:49.628769+0000","last_became_peered":"2024-03-20T08:45:49.628769+0000","last_unstale":"2024-03-20T08:46:29.605660+0000","last_undegraded":"2024-03-20T08:46:29.605660+0000","last_fullsized":"2024-03-20T08:46:29.605660+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-20T08:45:48.603868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:45:48.603868+0000","last_clean_scrub_stamp":"2024-03-20T08:45:48.603868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:30:36.931224+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":36,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-20T08:46:29.684117+0000","last_change":"2024-03-20T08:45:53.658005+0000","last_active":"2024-03-20T08:46:29.684117+0000","last_peered":"2024-03-20T08:46:29.684117+0000","last_clean":"2024-03-20T08:46:29.684117+0000","last_became_active":"2024-03-20T08:45:49.626638+0000","last_became_peered":"2024-03-20T08:45:49.626638+0000","last_unstale":"2024-03-20T08:46:29.684117+0000","last_undegraded":"2024-03-20T08:46:29.684117+0000","last_fullsized":"2024-03-20T08:46:29.684117+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-20T08:45:48.603868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:45:48.603868+0000","last_clean_scrub_stamp":"2024-03-20T08:45:48.603868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T09:43:03.860694+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0011759870000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"67'13","reported_seq":52,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-20T08:46:29.607988+0000","last_change":"2024-03-20T08:45:51.651666+0000","last_active":"2024-03-20T08:46:29.607988+0000","last_peered":"2024-03-20T08:46:29.607988+0000","last_clean":"2024-03-20T08:46:29.607988+0000","last_became_active":"2024-03-20T08:45:51.651322+0000","last_became_peered":"2024-03-20T08:45:51.651322+0000","last_unstale":"2024-03-20T08:46:29.607988+0000","last_undegraded":"2024-03-20T08:46:29.607988+0000","last_fullsized":"2024-03-20T08:46:29.607988+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-20T08:45:50.616760+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:45:50.616760+0000","last_clean_scrub_stamp":"2024-03-20T08:45:50.616760+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T19:58:27.420251+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":19,"num_write":16,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":21,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-20T08:46:29.684400+0000","last_change":"2024-03-20T08:45:53.666140+0000","last_active":"2024-03-20T08:46:29.684400+0000","last_peered":"2024-03-20T08:46:29.684400+0000","last_clean":"2024-03-20T08:46:29.684400+0000","last_became_active":"2024-03-20T08:45:53.665809+0000","last_became_peered":"2024-03-20T08:45:53.665809+0000","last_unstale":"2024-03-20T08:46:29.684400+0000","last_undegraded":"2024-03-20T08:46:29.684400+0000","last_fullsized":"2024-03-20T08:46:29.684400+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-20T08:45:52.630650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:45:52.630650+0000","last_clean_scrub_stamp":"2024-03-20T08:45:52.630650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T12:11:41.730622+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":17,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-20T08:46:29.607148+0000","last_change":"2024-03-20T08:45:55.691156+0000","last_active":"2024-03-20T08:46:29.607148+0000","last_peered":"2024-03-20T08:46:29.607148+0000","last_clean":"2024-03-20T08:46:29.607148+0000","last_became_active":"2024-03-20T08:45:55.690636+0000","last_became_peered":"2024-03-20T08:45:55.690636+0000","last_unstale":"2024-03-20T08:46:29.607148+0000","last_undegraded":"2024-03-20T08:46:29.607148+0000","last_fullsized":"2024-03-20T08:46:29.607148+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-20T08:45:54.654511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:45:54.654511+0000","last_clean_scrub_stamp":"2024-03-20T08:45:54.654511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T10:45:58.440667+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"69'188","reported_seq":337,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-20T08:46:31.703792+0000","last_change":"2024-03-20T08:45:35.514394+0000","last_active":"2024-03-20T08:46:31.703792+0000","last_peered":"2024-03-20T08:46:31.703792+0000","last_clean":"2024-03-20T08:46:31.703792+0000","last_became_active":"2024-03-20T08:45:35.513993+0000","last_became_peered":"2024-03-20T08:45:35.513993+0000","last_unstale":"2024-03-20T08:46:31.703792+0000","last_undegraded":"2024-03-20T08:46:31.703792+0000","last_fullsized":"2024-03-20T08:46:31.703792+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-20T08:43:22.115757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:43:22.115757+0000","last_clean_scrub_stamp":"2024-03-20T08:43:22.115757+0000","objects_scrubbed":0,"log_size":188,"log_dups_size":0,"ondisk_log_size":188,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T11:51:51.680611+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":242,"num_read_kb":333,"num_write":245,"num_write_kb":3982,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":29,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-20T08:46:29.686111+0000","last_change":"2024-03-20T08:45:49.633722+0000","last_active":"2024-03-20T08:46:29.686111+0000","last_peered":"2024-03-20T08:46:29.686111+0000","last_clean":"2024-03-20T08:46:29.686111+0000","last_became_active":"2024-03-20T08:45:49.633444+0000","last_became_peered":"2024-03-20T08:45:49.633444+0000","last_unstale":"2024-03-20T08:46:29.686111+0000","last_undegraded":"2024-03-20T08:46:29.686111+0000","last_fullsized":"2024-03-20T08:46:29.686111+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-20T08:45:48.603868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:45:48.603868+0000","last_clean_scrub_stamp":"2024-03-20T08:45:48.603868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T20:24:42.236163+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"67'16","reported_seq":65,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-20T08:47:09.686652+0000","last_change":"2024-03-20T08:45:51.645074+0000","last_active":"2024-03-20T08:47:09.686652+0000","last_peered":"2024-03-20T08:47:09.686652+0000","last_clean":"2024-03-20T08:47:09.686652+0000","last_became_active":"2024-03-20T08:45:51.644762+0000","last_became_peered":"2024-03-20T08:45:51.644762+0000","last_unstale":"2024-03-20T08:47:09.686652+0000","last_undegraded":"2024-03-20T08:47:09.686652+0000","last_fullsized":"2024-03-20T08:47:09.686652+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-20T08:45:50.616760+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:45:50.616760+0000","last_clean_scrub_stamp":"2024-03-20T08:45:50.616760+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T09:12:27.582168+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":25,"num_read_kb":15,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":21,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-20T08:46:29.683638+0000","last_change":"2024-03-20T08:45:53.666607+0000","last_active":"2024-03-20T08:46:29.683638+0000","last_peered":"2024-03-20T08:46:29.683638+0000","last_clean":"2024-03-20T08:46:29.683638+0000","last_became_active":"2024-03-20T08:45:53.666132+0000","last_became_peered":"2024-03-20T08:45:53.666132+0000","last_unstale":"2024-03-20T08:46:29.683638+0000","last_undegraded":"2024-03-20T08:46:29.683638+0000","last_fullsized":"2024-03-20T08:46:29.683638+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-20T08:45:52.630650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:45:52.630650+0000","last_clean_scrub_stamp":"2024-03-20T08:45:52.630650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T09:30:15.843001+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":17,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-20T08:46:29.606181+0000","last_change":"2024-03-20T08:45:55.691471+0000","last_active":"2024-03-20T08:46:29.606181+0000","last_peered":"2024-03-20T08:46:29.606181+0000","last_clean":"2024-03-20T08:46:29.606181+0000","last_became_active":"2024-03-20T08:45:55.690776+0000","last_became_peered":"2024-03-20T08:45:55.690776+0000","last_unstale":"2024-03-20T08:46:29.606181+0000","last_undegraded":"2024-03-20T08:46:29.606181+0000","last_fullsized":"2024-03-20T08:46:29.606181+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-20T08:45:54.654511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:45:54.654511+0000","last_clean_scrub_stamp":"2024-03-20T08:45:54.654511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:41:00.102045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":29,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-20T08:46:29.610362+0000","last_change":"2024-03-20T08:45:49.626166+0000","last_active":"2024-03-20T08:46:29.610362+0000","last_peered":"2024-03-20T08:46:29.610362+0000","last_clean":"2024-03-20T08:46:29.610362+0000","last_became_active":"2024-03-20T08:45:49.625892+0000","last_became_peered":"2024-03-20T08:45:49.625892+0000","last_unstale":"2024-03-20T08:46:29.610362+0000","last_undegraded":"2024-03-20T08:46:29.610362+0000","last_fullsized":"2024-03-20T08:46:29.610362+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-20T08:45:48.603868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:45:48.603868+0000","last_clean_scrub_stamp":"2024-03-20T08:45:48.603868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:14:03.125268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"68'5","reported_seq":119,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-20T08:47:14.012053+0000","last_change":"2024-03-20T08:45:53.658023+0000","last_active":"2024-03-20T08:47:14.012053+0000","last_peered":"2024-03-20T08:47:14.012053+0000","last_clean":"2024-03-20T08:47:14.012053+0000","last_became_active":"2024-03-20T08:45:49.634599+0000","last_became_peered":"2024-03-20T08:45:49.634599+0000","last_unstale":"2024-03-20T08:47:14.012053+0000","last_undegraded":"2024-03-20T08:47:14.012053+0000","last_fullsized":"2024-03-20T08:47:14.012053+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-20T08:45:48.603868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:45:48.603868+0000","last_clean_scrub_stamp":"2024-03-20T08:45:48.603868+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T12:27:18.966005+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0044392149999999998,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":80,"num_read_kb":75,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"67'12","reported_seq":43,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-20T08:46:29.610185+0000","last_change":"2024-03-20T08:45:51.649602+0000","last_active":"2024-03-20T08:46:29.610185+0000","last_peered":"2024-03-20T08:46:29.610185+0000","last_clean":"2024-03-20T08:46:29.610185+0000","last_became_active":"2024-03-20T08:45:51.649259+0000","last_became_peered":"2024-03-20T08:45:51.649259+0000","last_unstale":"2024-03-20T08:46:29.610185+0000","last_undegraded":"2024-03-20T08:46:29.610185+0000","last_fullsized":"2024-03-20T08:46:29.610185+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-20T08:45:50.616760+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:45:50.616760+0000","last_clean_scrub_stamp":"2024-03-20T08:45:50.616760+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T11:24:07.689854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":18,"num_read_kb":12,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":21,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-20T08:46:29.607394+0000","last_change":"2024-03-20T08:45:53.657977+0000","last_active":"2024-03-20T08:46:29.607394+0000","last_peered":"2024-03-20T08:46:29.607394+0000","last_clean":"2024-03-20T08:46:29.607394+0000","last_became_active":"2024-03-20T08:45:53.657640+0000","last_became_peered":"2024-03-20T08:45:53.657640+0000","last_unstale":"2024-03-20T08:46:29.607394+0000","last_undegraded":"2024-03-20T08:46:29.607394+0000","last_fullsized":"2024-03-20T08:46:29.607394+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-20T08:45:52.630650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:45:52.630650+0000","last_clean_scrub_stamp":"2024-03-20T08:45:52.630650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T08:46:00.972890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1","version":"67'1","reported_seq":18,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-20T08:46:29.605533+0000","last_change":"2024-03-20T08:45:55.815254+0000","last_active":"2024-03-20T08:46:29.605533+0000","last_peered":"2024-03-20T08:46:29.605533+0000","last_clean":"2024-03-20T08:46:29.605533+0000","last_became_active":"2024-03-20T08:45:55.814810+0000","last_became_peered":"2024-03-20T08:45:55.814810+0000","last_unstale":"2024-03-20T08:46:29.605533+0000","last_undegraded":"2024-03-20T08:46:29.605533+0000","last_fullsized":"2024-03-20T08:46:29.605533+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-20T08:45:54.654511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:45:54.654511+0000","last_clean_scrub_stamp":"2024-03-20T08:45:54.654511+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T18:39:56.308158+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":29,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-20T08:46:29.608270+0000","last_change":"2024-03-20T08:45:49.627368+0000","last_active":"2024-03-20T08:46:29.608270+0000","last_peered":"2024-03-20T08:46:29.608270+0000","last_clean":"2024-03-20T08:46:29.608270+0000","last_became_active":"2024-03-20T08:45:49.627035+0000","last_became_peered":"2024-03-20T08:45:49.627035+0000","last_unstale":"2024-03-20T08:46:29.608270+0000","last_undegraded":"2024-03-20T08:46:29.608270+0000","last_fullsized":"2024-03-20T08:46:29.608270+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-20T08:45:48.603868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:45:48.603868+0000","last_clean_scrub_stamp":"2024-03-20T08:45:48.603868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T19:58:15.422219+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.3","version":"67'19","reported_seq":61,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-20T08:46:29.611153+0000","last_change":"2024-03-20T08:45:51.642189+0000","last_active":"2024-03-20T08:46:29.611153+0000","last_peered":"2024-03-20T08:46:29.611153+0000","last_clean":"2024-03-20T08:46:29.611153+0000","last_became_active":"2024-03-20T08:45:51.641430+0000","last_became_peered":"2024-03-20T08:45:51.641430+0000","last_unstale":"2024-03-20T08:46:29.611153+0000","last_undegraded":"2024-03-20T08:46:29.611153+0000","last_fullsized":"2024-03-20T08:46:29.611153+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-20T08:45:50.616760+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:45:50.616760+0000","last_clean_scrub_stamp":"2024-03-20T08:45:50.616760+0000","objects_scrubbed":0,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T10:10:41.927770+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":39,"num_read_kb":25,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":21,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-20T08:46:29.684204+0000","last_change":"2024-03-20T08:45:53.660344+0000","last_active":"2024-03-20T08:46:29.684204+0000","last_peered":"2024-03-20T08:46:29.684204+0000","last_clean":"2024-03-20T08:46:29.684204+0000","last_became_active":"2024-03-20T08:45:53.660035+0000","last_became_peered":"2024-03-20T08:45:53.660035+0000","last_unstale":"2024-03-20T08:46:29.684204+0000","last_undegraded":"2024-03-20T08:46:29.684204+0000","last_fullsized":"2024-03-20T08:46:29.684204+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-20T08:45:52.630650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:45:52.630650+0000","last_clean_scrub_stamp":"2024-03-20T08:45:52.630650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:39:37.660656+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":17,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-20T08:46:29.609823+0000","last_change":"2024-03-20T08:45:55.693872+0000","last_active":"2024-03-20T08:46:29.609823+0000","last_peered":"2024-03-20T08:46:29.609823+0000","last_clean":"2024-03-20T08:46:29.609823+0000","last_became_active":"2024-03-20T08:45:55.693399+0000","last_became_peered":"2024-03-20T08:45:55.693399+0000","last_unstale":"2024-03-20T08:46:29.609823+0000","last_undegraded":"2024-03-20T08:46:29.609823+0000","last_fullsized":"2024-03-20T08:46:29.609823+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-20T08:45:54.654511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:45:54.654511+0000","last_clean_scrub_stamp":"2024-03-20T08:45:54.654511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T12:05:11.432672+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":29,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-20T08:46:29.607440+0000","last_change":"2024-03-20T08:45:49.631145+0000","last_active":"2024-03-20T08:46:29.607440+0000","last_peered":"2024-03-20T08:46:29.607440+0000","last_clean":"2024-03-20T08:46:29.607440+0000","last_became_active":"2024-03-20T08:45:49.630851+0000","last_became_peered":"2024-03-20T08:45:49.630851+0000","last_unstale":"2024-03-20T08:46:29.607440+0000","last_undegraded":"2024-03-20T08:46:29.607440+0000","last_fullsized":"2024-03-20T08:46:29.607440+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-20T08:45:48.603868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:45:48.603868+0000","last_clean_scrub_stamp":"2024-03-20T08:45:48.603868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T17:18:59.598326+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.2","version":"67'10","reported_seq":40,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-20T08:46:29.606339+0000","last_change":"2024-03-20T08:45:51.647715+0000","last_active":"2024-03-20T08:46:29.606339+0000","last_peered":"2024-03-20T08:46:29.606339+0000","last_clean":"2024-03-20T08:46:29.606339+0000","last_became_active":"2024-03-20T08:45:51.647382+0000","last_became_peered":"2024-03-20T08:45:51.647382+0000","last_unstale":"2024-03-20T08:46:29.606339+0000","last_undegraded":"2024-03-20T08:46:29.606339+0000","last_fullsized":"2024-03-20T08:46:29.606339+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-20T08:45:50.616760+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:45:50.616760+0000","last_clean_scrub_stamp":"2024-03-20T08:45:50.616760+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T12:14:30.581405+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3","version":"68'11","reported_seq":50,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-20T08:47:09.686651+0000","last_change":"2024-03-20T08:45:53.658746+0000","last_active":"2024-03-20T08:47:09.686651+0000","last_peered":"2024-03-20T08:47:09.686651+0000","last_clean":"2024-03-20T08:47:09.686651+0000","last_became_active":"2024-03-20T08:45:53.657489+0000","last_became_peered":"2024-03-20T08:45:53.657489+0000","last_unstale":"2024-03-20T08:47:09.686651+0000","last_undegraded":"2024-03-20T08:47:09.686651+0000","last_fullsized":"2024-03-20T08:47:09.686651+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-20T08:45:52.630650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:45:52.630650+0000","last_clean_scrub_stamp":"2024-03-20T08:45:52.630650+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T20:13:54.527052+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":17,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-20T08:46:29.684056+0000","last_change":"2024-03-20T08:45:55.817446+0000","last_active":"2024-03-20T08:46:29.684056+0000","last_peered":"2024-03-20T08:46:29.684056+0000","last_clean":"2024-03-20T08:46:29.684056+0000","last_became_active":"2024-03-20T08:45:55.817107+0000","last_became_peered":"2024-03-20T08:45:55.817107+0000","last_unstale":"2024-03-20T08:46:29.684056+0000","last_undegraded":"2024-03-20T08:46:29.684056+0000","last_fullsized":"2024-03-20T08:46:29.684056+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-20T08:45:54.654511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:45:54.654511+0000","last_clean_scrub_stamp":"2024-03-20T08:45:54.654511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T18:24:02.978084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.6","version":"60'1","reported_seq":30,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-20T08:46:29.610310+0000","last_change":"2024-03-20T08:45:49.623198+0000","last_active":"2024-03-20T08:46:29.610310+0000","last_peered":"2024-03-20T08:46:29.610310+0000","last_clean":"2024-03-20T08:46:29.610310+0000","last_became_active":"2024-03-20T08:45:49.622878+0000","last_became_peered":"2024-03-20T08:45:49.622878+0000","last_unstale":"2024-03-20T08:46:29.610310+0000","last_undegraded":"2024-03-20T08:46:29.610310+0000","last_fullsized":"2024-03-20T08:46:29.610310+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-20T08:45:48.603868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:45:48.603868+0000","last_clean_scrub_stamp":"2024-03-20T08:45:48.603868+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T12:47:16.777271+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1","version":"67'14","reported_seq":46,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-20T08:46:29.658705+0000","last_change":"2024-03-20T08:45:51.643180+0000","last_active":"2024-03-20T08:46:29.658705+0000","last_peered":"2024-03-20T08:46:29.658705+0000","last_clean":"2024-03-20T08:46:29.658705+0000","last_became_active":"2024-03-20T08:45:51.642854+0000","last_became_peered":"2024-03-20T08:45:51.642854+0000","last_unstale":"2024-03-20T08:46:29.658705+0000","last_undegraded":"2024-03-20T08:46:29.658705+0000","last_fullsized":"2024-03-20T08:46:29.658705+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-20T08:45:50.616760+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:45:50.616760+0000","last_clean_scrub_stamp":"2024-03-20T08:45:50.616760+0000","objects_scrubbed":0,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T12:52:53.657307+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":21,"num_read_kb":14,"num_write":14,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.0","version":"68'11","reported_seq":50,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-20T08:47:09.686642+0000","last_change":"2024-03-20T08:45:53.659549+0000","last_active":"2024-03-20T08:47:09.686642+0000","last_peered":"2024-03-20T08:47:09.686642+0000","last_clean":"2024-03-20T08:47:09.686642+0000","last_became_active":"2024-03-20T08:45:53.659203+0000","last_became_peered":"2024-03-20T08:45:53.659203+0000","last_unstale":"2024-03-20T08:47:09.686642+0000","last_undegraded":"2024-03-20T08:47:09.686642+0000","last_fullsized":"2024-03-20T08:47:09.686642+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-20T08:45:52.630650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:45:52.630650+0000","last_clean_scrub_stamp":"2024-03-20T08:45:52.630650+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T15:43:37.857631+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":17,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-20T08:46:29.658705+0000","last_change":"2024-03-20T08:45:55.690584+0000","last_active":"2024-03-20T08:46:29.658705+0000","last_peered":"2024-03-20T08:46:29.658705+0000","last_clean":"2024-03-20T08:46:29.658705+0000","last_became_active":"2024-03-20T08:45:55.690284+0000","last_became_peered":"2024-03-20T08:45:55.690284+0000","last_unstale":"2024-03-20T08:46:29.658705+0000","last_undegraded":"2024-03-20T08:46:29.658705+0000","last_fullsized":"2024-03-20T08:46:29.658705+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-20T08:45:54.654511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:45:54.654511+0000","last_clean_scrub_stamp":"2024-03-20T08:45:54.654511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T09:01:02.410099+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":29,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-20T08:46:29.685984+0000","last_change":"2024-03-20T08:45:49.633279+0000","last_active":"2024-03-20T08:46:29.685984+0000","last_peered":"2024-03-20T08:46:29.685984+0000","last_clean":"2024-03-20T08:46:29.685984+0000","last_became_active":"2024-03-20T08:45:49.633000+0000","last_became_peered":"2024-03-20T08:45:49.633000+0000","last_unstale":"2024-03-20T08:46:29.685984+0000","last_undegraded":"2024-03-20T08:46:29.685984+0000","last_fullsized":"2024-03-20T08:46:29.685984+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-20T08:45:48.603868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:45:48.603868+0000","last_clean_scrub_stamp":"2024-03-20T08:45:48.603868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T20:20:55.450167+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.0","version":"67'18","reported_seq":57,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-20T08:46:29.686025+0000","last_change":"2024-03-20T08:45:51.648290+0000","last_active":"2024-03-20T08:46:29.686025+0000","last_peered":"2024-03-20T08:46:29.686025+0000","last_clean":"2024-03-20T08:46:29.686025+0000","last_became_active":"2024-03-20T08:45:51.647996+0000","last_became_peered":"2024-03-20T08:45:51.647996+0000","last_unstale":"2024-03-20T08:46:29.686025+0000","last_undegraded":"2024-03-20T08:46:29.686025+0000","last_fullsized":"2024-03-20T08:46:29.686025+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-20T08:45:50.616760+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:45:50.616760+0000","last_clean_scrub_stamp":"2024-03-20T08:45:50.616760+0000","objects_scrubbed":0,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T11:09:28.409992+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":34,"num_read_kb":22,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":21,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-20T08:46:29.658565+0000","last_change":"2024-03-20T08:45:53.662892+0000","last_active":"2024-03-20T08:46:29.658565+0000","last_peered":"2024-03-20T08:46:29.658565+0000","last_clean":"2024-03-20T08:46:29.658565+0000","last_became_active":"2024-03-20T08:45:53.662308+0000","last_became_peered":"2024-03-20T08:45:53.662308+0000","last_unstale":"2024-03-20T08:46:29.658565+0000","last_undegraded":"2024-03-20T08:46:29.658565+0000","last_fullsized":"2024-03-20T08:46:29.658565+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-20T08:45:52.630650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:45:52.630650+0000","last_clean_scrub_stamp":"2024-03-20T08:45:52.630650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T15:50:00.452961+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":17,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-20T08:46:29.606822+0000","last_change":"2024-03-20T08:45:55.687291+0000","last_active":"2024-03-20T08:46:29.606822+0000","last_peered":"2024-03-20T08:46:29.606822+0000","last_clean":"2024-03-20T08:46:29.606822+0000","last_became_active":"2024-03-20T08:45:55.687016+0000","last_became_peered":"2024-03-20T08:45:55.687016+0000","last_unstale":"2024-03-20T08:46:29.606822+0000","last_undegraded":"2024-03-20T08:46:29.606822+0000","last_fullsized":"2024-03-20T08:46:29.606822+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-20T08:45:54.654511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:45:54.654511+0000","last_clean_scrub_stamp":"2024-03-20T08:45:54.654511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T20:15:41.744529+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.e","version":"68'11","reported_seq":50,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-20T08:47:09.686761+0000","last_change":"2024-03-20T08:45:53.655202+0000","last_active":"2024-03-20T08:47:09.686761+0000","last_peered":"2024-03-20T08:47:09.686761+0000","last_clean":"2024-03-20T08:47:09.686761+0000","last_became_active":"2024-03-20T08:45:53.654874+0000","last_became_peered":"2024-03-20T08:45:53.654874+0000","last_unstale":"2024-03-20T08:47:09.686761+0000","last_undegraded":"2024-03-20T08:47:09.686761+0000","last_fullsized":"2024-03-20T08:47:09.686761+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-20T08:45:52.630650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:45:52.630650+0000","last_clean_scrub_stamp":"2024-03-20T08:45:52.630650+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T11:08:48.289832+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":29,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-20T08:46:29.685614+0000","last_change":"2024-03-20T08:45:49.628572+0000","last_active":"2024-03-20T08:46:29.685614+0000","last_peered":"2024-03-20T08:46:29.685614+0000","last_clean":"2024-03-20T08:46:29.685614+0000","last_became_active":"2024-03-20T08:45:49.628236+0000","last_became_peered":"2024-03-20T08:45:49.628236+0000","last_unstale":"2024-03-20T08:46:29.685614+0000","last_undegraded":"2024-03-20T08:46:29.685614+0000","last_fullsized":"2024-03-20T08:46:29.685614+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-20T08:45:48.603868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:45:48.603868+0000","last_clean_scrub_stamp":"2024-03-20T08:45:48.603868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:42:36.383174+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"67'15","reported_seq":50,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-20T08:46:29.607854+0000","last_change":"2024-03-20T08:45:51.649560+0000","last_active":"2024-03-20T08:46:29.607854+0000","last_peered":"2024-03-20T08:46:29.607854+0000","last_clean":"2024-03-20T08:46:29.607854+0000","last_became_active":"2024-03-20T08:45:51.649096+0000","last_became_peered":"2024-03-20T08:45:51.649096+0000","last_unstale":"2024-03-20T08:46:29.607854+0000","last_undegraded":"2024-03-20T08:46:29.607854+0000","last_fullsized":"2024-03-20T08:46:29.607854+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-20T08:45:50.616760+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:45:50.616760+0000","last_clean_scrub_stamp":"2024-03-20T08:45:50.616760+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T10:10:45.711033+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":17,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-20T08:46:29.686902+0000","last_change":"2024-03-20T08:45:55.816439+0000","last_active":"2024-03-20T08:46:29.686902+0000","last_peered":"2024-03-20T08:46:29.686902+0000","last_clean":"2024-03-20T08:46:29.686902+0000","last_became_active":"2024-03-20T08:45:55.814953+0000","last_became_peered":"2024-03-20T08:45:55.814953+0000","last_unstale":"2024-03-20T08:46:29.686902+0000","last_undegraded":"2024-03-20T08:46:29.686902+0000","last_fullsized":"2024-03-20T08:46:29.686902+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-20T08:45:54.654511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:45:54.654511+0000","last_clean_scrub_stamp":"2024-03-20T08:45:54.654511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T12:34:38.123915+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":21,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-20T08:46:29.607337+0000","last_change":"2024-03-20T08:45:53.665023+0000","last_active":"2024-03-20T08:46:29.607337+0000","last_peered":"2024-03-20T08:46:29.607337+0000","last_clean":"2024-03-20T08:46:29.607337+0000","last_became_active":"2024-03-20T08:45:53.664715+0000","last_became_peered":"2024-03-20T08:45:53.664715+0000","last_unstale":"2024-03-20T08:46:29.607337+0000","last_undegraded":"2024-03-20T08:46:29.607337+0000","last_fullsized":"2024-03-20T08:46:29.607337+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-20T08:45:52.630650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:45:52.630650+0000","last_clean_scrub_stamp":"2024-03-20T08:45:52.630650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T12:28:35.702350+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":29,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-20T08:46:29.658713+0000","last_change":"2024-03-20T08:45:49.631944+0000","last_active":"2024-03-20T08:46:29.658713+0000","last_peered":"2024-03-20T08:46:29.658713+0000","last_clean":"2024-03-20T08:46:29.658713+0000","last_became_active":"2024-03-20T08:45:49.629022+0000","last_became_peered":"2024-03-20T08:45:49.629022+0000","last_unstale":"2024-03-20T08:46:29.658713+0000","last_undegraded":"2024-03-20T08:46:29.658713+0000","last_fullsized":"2024-03-20T08:46:29.658713+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-20T08:45:48.603868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:45:48.603868+0000","last_clean_scrub_stamp":"2024-03-20T08:45:48.603868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:28:00.389627+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"67'11","reported_seq":44,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-20T08:46:29.659311+0000","last_change":"2024-03-20T08:45:51.648565+0000","last_active":"2024-03-20T08:46:29.659311+0000","last_peered":"2024-03-20T08:46:29.659311+0000","last_clean":"2024-03-20T08:46:29.659311+0000","last_became_active":"2024-03-20T08:45:51.648255+0000","last_became_peered":"2024-03-20T08:45:51.648255+0000","last_unstale":"2024-03-20T08:46:29.659311+0000","last_undegraded":"2024-03-20T08:46:29.659311+0000","last_fullsized":"2024-03-20T08:46:29.659311+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-20T08:45:50.616760+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:45:50.616760+0000","last_clean_scrub_stamp":"2024-03-20T08:45:50.616760+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T15:47:01.342506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":17,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-20T08:46:29.684706+0000","last_change":"2024-03-20T08:45:55.690766+0000","last_active":"2024-03-20T08:46:29.684706+0000","last_peered":"2024-03-20T08:46:29.684706+0000","last_clean":"2024-03-20T08:46:29.684706+0000","last_became_active":"2024-03-20T08:45:55.690354+0000","last_became_peered":"2024-03-20T08:45:55.690354+0000","last_unstale":"2024-03-20T08:46:29.684706+0000","last_undegraded":"2024-03-20T08:46:29.684706+0000","last_fullsized":"2024-03-20T08:46:29.684706+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-20T08:45:54.654511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:45:54.654511+0000","last_clean_scrub_stamp":"2024-03-20T08:45:54.654511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T08:58:43.782547+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":21,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-20T08:46:29.608549+0000","last_change":"2024-03-20T08:45:53.662905+0000","last_active":"2024-03-20T08:46:29.608549+0000","last_peered":"2024-03-20T08:46:29.608549+0000","last_clean":"2024-03-20T08:46:29.608549+0000","last_became_active":"2024-03-20T08:45:53.662390+0000","last_became_peered":"2024-03-20T08:45:53.662390+0000","last_unstale":"2024-03-20T08:46:29.608549+0000","last_undegraded":"2024-03-20T08:46:29.608549+0000","last_fullsized":"2024-03-20T08:46:29.608549+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-20T08:45:52.630650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:45:52.630650+0000","last_clean_scrub_stamp":"2024-03-20T08:45:52.630650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:08:02.985347+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":29,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-20T08:46:29.683344+0000","last_change":"2024-03-20T08:45:49.627755+0000","last_active":"2024-03-20T08:46:29.683344+0000","last_peered":"2024-03-20T08:46:29.683344+0000","last_clean":"2024-03-20T08:46:29.683344+0000","last_became_active":"2024-03-20T08:45:49.627529+0000","last_became_peered":"2024-03-20T08:45:49.627529+0000","last_unstale":"2024-03-20T08:46:29.683344+0000","last_undegraded":"2024-03-20T08:46:29.683344+0000","last_fullsized":"2024-03-20T08:46:29.683344+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-20T08:45:48.603868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:45:48.603868+0000","last_clean_scrub_stamp":"2024-03-20T08:45:48.603868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T16:56:35.899277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.d","version":"67'17","reported_seq":53,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-20T08:46:29.659327+0000","last_change":"2024-03-20T08:45:51.653119+0000","last_active":"2024-03-20T08:46:29.659327+0000","last_peered":"2024-03-20T08:46:29.659327+0000","last_clean":"2024-03-20T08:46:29.659327+0000","last_became_active":"2024-03-20T08:45:51.652808+0000","last_became_peered":"2024-03-20T08:45:51.652808+0000","last_unstale":"2024-03-20T08:46:29.659327+0000","last_undegraded":"2024-03-20T08:46:29.659327+0000","last_fullsized":"2024-03-20T08:46:29.659327+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-20T08:45:50.616760+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:45:50.616760+0000","last_clean_scrub_stamp":"2024-03-20T08:45:50.616760+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T10:13:15.965338+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":19,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":17,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-20T08:46:29.659501+0000","last_change":"2024-03-20T08:45:55.687136+0000","last_active":"2024-03-20T08:46:29.659501+0000","last_peered":"2024-03-20T08:46:29.659501+0000","last_clean":"2024-03-20T08:46:29.659501+0000","last_became_active":"2024-03-20T08:45:55.686739+0000","last_became_peered":"2024-03-20T08:45:55.686739+0000","last_unstale":"2024-03-20T08:46:29.659501+0000","last_undegraded":"2024-03-20T08:46:29.659501+0000","last_fullsized":"2024-03-20T08:46:29.659501+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-20T08:45:54.654511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:45:54.654511+0000","last_clean_scrub_stamp":"2024-03-20T08:45:54.654511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T20:00:48.398479+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"68'11","reported_seq":53,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-20T08:47:09.686663+0000","last_change":"2024-03-20T08:45:53.666994+0000","last_active":"2024-03-20T08:47:09.686663+0000","last_peered":"2024-03-20T08:47:09.686663+0000","last_clean":"2024-03-20T08:47:09.686663+0000","last_became_active":"2024-03-20T08:45:53.666656+0000","last_became_peered":"2024-03-20T08:45:53.666656+0000","last_unstale":"2024-03-20T08:47:09.686663+0000","last_undegraded":"2024-03-20T08:47:09.686663+0000","last_fullsized":"2024-03-20T08:47:09.686663+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-20T08:45:52.630650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:45:52.630650+0000","last_clean_scrub_stamp":"2024-03-20T08:45:52.630650+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:37:42.431488+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":29,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-20T08:46:29.686564+0000","last_change":"2024-03-20T08:45:49.630406+0000","last_active":"2024-03-20T08:46:29.686564+0000","last_peered":"2024-03-20T08:46:29.686564+0000","last_clean":"2024-03-20T08:46:29.686564+0000","last_became_active":"2024-03-20T08:45:49.630108+0000","last_became_peered":"2024-03-20T08:45:49.630108+0000","last_unstale":"2024-03-20T08:46:29.686564+0000","last_undegraded":"2024-03-20T08:46:29.686564+0000","last_fullsized":"2024-03-20T08:46:29.686564+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-20T08:45:48.603868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:45:48.603868+0000","last_clean_scrub_stamp":"2024-03-20T08:45:48.603868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:17:21.499429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"67'10","reported_seq":40,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-20T08:46:29.659096+0000","last_change":"2024-03-20T08:45:51.646151+0000","last_active":"2024-03-20T08:46:29.659096+0000","last_peered":"2024-03-20T08:46:29.659096+0000","last_clean":"2024-03-20T08:46:29.659096+0000","last_became_active":"2024-03-20T08:45:51.645785+0000","last_became_peered":"2024-03-20T08:45:51.645785+0000","last_unstale":"2024-03-20T08:46:29.659096+0000","last_undegraded":"2024-03-20T08:46:29.659096+0000","last_fullsized":"2024-03-20T08:46:29.659096+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-20T08:45:50.616760+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:45:50.616760+0000","last_clean_scrub_stamp":"2024-03-20T08:45:50.616760+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T20:03:40.452309+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":383,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1149,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":88,"ondisk_log_size":88,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":701,"num_read_kb":458,"num_write":417,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":395,"ondisk_log_size":395,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1531,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":34,"num_read_kb":34,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4593,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":96,"acting":96,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":242,"num_read_kb":333,"num_write":245,"num_write_kb":3982,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":188,"ondisk_log_size":188,"up":3,"acting":3,"num_store_stats":3},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":80,"num_read_kb":75,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":9,"acting":9,"num_store_stats":7}],"osd_stats":[{"osd":7,"up_from":54,"seq":231928234007,"num_pgs":53,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93741056,"kb_used":32516,"kb_used_data":1340,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93708540,"statfs":{"total":95990841344,"available":95957544960,"internally_reserved":0,"allocated":1372160,"data_stored":909424,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":46,"seq":197568495644,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93741056,"kb_used":32492,"kb_used_data":1316,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93708564,"statfs":{"total":95990841344,"available":95957569536,"internally_reserved":0,"allocated":1347584,"data_stored":907038,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574902,"num_pgs":57,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93741056,"kb_used":31984,"kb_used_data":804,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93709072,"statfs":{"total":95990841344,"available":95958089728,"internally_reserved":0,"allocated":823296,"data_stored":319366,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738428,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93741056,"kb_used":28352,"kb_used_data":1336,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93712704,"statfs":{"total":95990841344,"available":95961808896,"internally_reserved":0,"allocated":1368064,"data_stored":907819,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411378,"num_pgs":36,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93741056,"kb_used":27752,"kb_used_data":732,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93713304,"statfs":{"total":95990841344,"available":95962423296,"internally_reserved":0,"allocated":749568,"data_stored":316785,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182444,"num_pgs":56,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93741056,"kb_used":31952,"kb_used_data":772,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93709104,"statfs":{"total":95990841344,"available":95958122496,"internally_reserved":0,"allocated":790528,"data_stored":317984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":30,"seq":128849018918,"num_pgs":58,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93741056,"kb_used":27784,"kb_used_data":764,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93713272,"statfs":{"total":95990841344,"available":95962390528,"internally_reserved":0,"allocated":782336,"data_stored":317698,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757281,"num_pgs":47,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93741056,"kb_used":31920,"kb_used_data":740,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93709136,"statfs":{"total":95990841344,"available":95958155264,"internally_reserved":0,"allocated":757760,"data_stored":317894,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":138,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1003,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":482,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":620,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":911,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1439,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":90112,"data_stored":2338,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":798,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":990,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-20T08:47:19.900 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 4c3f3eba-e695-11ee-95c9-87774f69a715 -- ceph pg dump --format=json 2024-03-20T08:47:20.065 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:47:19 smithi194 ceph-mon[27237]: from='client.24674 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-20T08:47:20.139 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/4c3f3eba-e695-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T08:47:20.205 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:47:19 smithi183 ceph-mon[25285]: from='client.24674 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-20T08:47:20.205 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:47:19 smithi183 ceph-mon[31860]: from='client.24674 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-20T08:47:20.888 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:47:20 smithi194 ceph-mon[27237]: pgmap v28: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-20T08:47:20.954 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:47:20 smithi183 ceph-mon[25285]: pgmap v28: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-20T08:47:20.955 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:47:20 smithi183 ceph-mon[31860]: pgmap v28: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-20T08:47:21.332 INFO:journalctl@ceph.iscsi.iscsi.a.smithi194.stdout:Mar 20 08:47:20 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-iscsi-iscsi-a[47467]: debug there is no tcmu-runner data available 2024-03-20T08:47:21.954 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:47:21 smithi183 ceph-mon[25285]: from='client.14559 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-20T08:47:21.955 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:47:21 smithi183 ceph-mon[31860]: from='client.14559 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-20T08:47:22.076 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-20T08:47:22.078 INFO:teuthology.orchestra.run.smithi183.stderr:dumped all 2024-03-20T08:47:22.138 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:47:21 smithi194 ceph-mon[27237]: from='client.14559 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-20T08:47:22.795 INFO:teuthology.orchestra.run.smithi183.stdout:{"pg_ready":true,"pg_map":{"version":29,"stamp":"2024-03-20T08:47:21.643208+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":596392,"num_objects":198,"num_object_clones":0,"num_object_copies":594,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":198,"num_whiteouts":0,"num_read":1065,"num_read_kb":908,"num_write":681,"num_write_kb":4027,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":686,"ondisk_log_size":686,"up":396,"acting":396,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":396,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749928448,"kb_used":244752,"kb_used_data":7804,"kb_used_omap":12,"kb_used_meta":236851,"kb_avail":749683696,"statfs":{"total":767926730752,"available":767676104704,"internally_reserved":0,"allocated":7991296,"data_stored":4314008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":242535984},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":15,"num_read_kb":15,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.016734"},"pg_stats":[{"pgid":"6.1a","version":"0'0","reported_seq":17,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-20T08:46:29.658663+0000","last_change":"2024-03-20T08:45:55.683275+0000","last_active":"2024-03-20T08:46:29.658663+0000","last_peered":"2024-03-20T08:46:29.658663+0000","last_clean":"2024-03-20T08:46:29.658663+0000","last_became_active":"2024-03-20T08:45:55.682979+0000","last_became_peered":"2024-03-20T08:45:55.682979+0000","last_unstale":"2024-03-20T08:46:29.658663+0000","last_undegraded":"2024-03-20T08:46:29.658663+0000","last_fullsized":"2024-03-20T08:46:29.658663+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-20T08:45:54.654511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:45:54.654511+0000","last_clean_scrub_stamp":"2024-03-20T08:45:54.654511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:58:05.609841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":21,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-20T08:46:29.605562+0000","last_change":"2024-03-20T08:45:53.657515+0000","last_active":"2024-03-20T08:46:29.605562+0000","last_peered":"2024-03-20T08:46:29.605562+0000","last_clean":"2024-03-20T08:46:29.605562+0000","last_became_active":"2024-03-20T08:45:53.657201+0000","last_became_peered":"2024-03-20T08:45:53.657201+0000","last_unstale":"2024-03-20T08:46:29.605562+0000","last_undegraded":"2024-03-20T08:46:29.605562+0000","last_fullsized":"2024-03-20T08:46:29.605562+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-20T08:45:52.630650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:45:52.630650+0000","last_clean_scrub_stamp":"2024-03-20T08:45:52.630650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T18:04:36.348891+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":29,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-20T08:46:29.611418+0000","last_change":"2024-03-20T08:45:49.627677+0000","last_active":"2024-03-20T08:46:29.611418+0000","last_peered":"2024-03-20T08:46:29.611418+0000","last_clean":"2024-03-20T08:46:29.611418+0000","last_became_active":"2024-03-20T08:45:49.627383+0000","last_became_peered":"2024-03-20T08:45:49.627383+0000","last_unstale":"2024-03-20T08:46:29.611418+0000","last_undegraded":"2024-03-20T08:46:29.611418+0000","last_fullsized":"2024-03-20T08:46:29.611418+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-20T08:45:48.603868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:45:48.603868+0000","last_clean_scrub_stamp":"2024-03-20T08:45:48.603868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:27:10.134498+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"67'9","reported_seq":41,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-20T08:46:29.658539+0000","last_change":"2024-03-20T08:45:51.652872+0000","last_active":"2024-03-20T08:46:29.658539+0000","last_peered":"2024-03-20T08:46:29.658539+0000","last_clean":"2024-03-20T08:46:29.658539+0000","last_became_active":"2024-03-20T08:45:51.652583+0000","last_became_peered":"2024-03-20T08:45:51.652583+0000","last_unstale":"2024-03-20T08:46:29.658539+0000","last_undegraded":"2024-03-20T08:46:29.658539+0000","last_fullsized":"2024-03-20T08:46:29.658539+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-20T08:45:50.616760+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:45:50.616760+0000","last_clean_scrub_stamp":"2024-03-20T08:45:50.616760+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T19:08:28.230482+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":17,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-20T08:46:29.686849+0000","last_change":"2024-03-20T08:45:55.816915+0000","last_active":"2024-03-20T08:46:29.686849+0000","last_peered":"2024-03-20T08:46:29.686849+0000","last_clean":"2024-03-20T08:46:29.686849+0000","last_became_active":"2024-03-20T08:45:55.816614+0000","last_became_peered":"2024-03-20T08:45:55.816614+0000","last_unstale":"2024-03-20T08:46:29.686849+0000","last_undegraded":"2024-03-20T08:46:29.686849+0000","last_fullsized":"2024-03-20T08:46:29.686849+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-20T08:45:54.654511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:45:54.654511+0000","last_clean_scrub_stamp":"2024-03-20T08:45:54.654511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T15:47:20.447708+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":21,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-20T08:46:29.658586+0000","last_change":"2024-03-20T08:45:53.657954+0000","last_active":"2024-03-20T08:46:29.658586+0000","last_peered":"2024-03-20T08:46:29.658586+0000","last_clean":"2024-03-20T08:46:29.658586+0000","last_became_active":"2024-03-20T08:45:53.657651+0000","last_became_peered":"2024-03-20T08:45:53.657651+0000","last_unstale":"2024-03-20T08:46:29.658586+0000","last_undegraded":"2024-03-20T08:46:29.658586+0000","last_fullsized":"2024-03-20T08:46:29.658586+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-20T08:45:52.630650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:45:52.630650+0000","last_clean_scrub_stamp":"2024-03-20T08:45:52.630650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T12:46:14.847461+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":29,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-20T08:46:29.687374+0000","last_change":"2024-03-20T08:45:49.628867+0000","last_active":"2024-03-20T08:46:29.687374+0000","last_peered":"2024-03-20T08:46:29.687374+0000","last_clean":"2024-03-20T08:46:29.687374+0000","last_became_active":"2024-03-20T08:45:49.628599+0000","last_became_peered":"2024-03-20T08:45:49.628599+0000","last_unstale":"2024-03-20T08:46:29.687374+0000","last_undegraded":"2024-03-20T08:46:29.687374+0000","last_fullsized":"2024-03-20T08:46:29.687374+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-20T08:45:48.603868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:45:48.603868+0000","last_clean_scrub_stamp":"2024-03-20T08:45:48.603868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T10:18:06.481115+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"67'15","reported_seq":50,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-20T08:46:29.685214+0000","last_change":"2024-03-20T08:45:51.646981+0000","last_active":"2024-03-20T08:46:29.685214+0000","last_peered":"2024-03-20T08:46:29.685214+0000","last_clean":"2024-03-20T08:46:29.685214+0000","last_became_active":"2024-03-20T08:45:51.646639+0000","last_became_peered":"2024-03-20T08:45:51.646639+0000","last_unstale":"2024-03-20T08:46:29.685214+0000","last_undegraded":"2024-03-20T08:46:29.685214+0000","last_fullsized":"2024-03-20T08:46:29.685214+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-20T08:45:50.616760+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:45:50.616760+0000","last_clean_scrub_stamp":"2024-03-20T08:45:50.616760+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T10:10:37.794721+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":17,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-20T08:46:29.609925+0000","last_change":"2024-03-20T08:45:55.682917+0000","last_active":"2024-03-20T08:46:29.609925+0000","last_peered":"2024-03-20T08:46:29.609925+0000","last_clean":"2024-03-20T08:46:29.609925+0000","last_became_active":"2024-03-20T08:45:55.682590+0000","last_became_peered":"2024-03-20T08:45:55.682590+0000","last_unstale":"2024-03-20T08:46:29.609925+0000","last_undegraded":"2024-03-20T08:46:29.609925+0000","last_fullsized":"2024-03-20T08:46:29.609925+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-20T08:45:54.654511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:45:54.654511+0000","last_clean_scrub_stamp":"2024-03-20T08:45:54.654511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:46:35.399856+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":21,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-20T08:46:29.607140+0000","last_change":"2024-03-20T08:45:53.657201+0000","last_active":"2024-03-20T08:46:29.607140+0000","last_peered":"2024-03-20T08:46:29.607140+0000","last_clean":"2024-03-20T08:46:29.607140+0000","last_became_active":"2024-03-20T08:45:53.656878+0000","last_became_peered":"2024-03-20T08:45:53.656878+0000","last_unstale":"2024-03-20T08:46:29.607140+0000","last_undegraded":"2024-03-20T08:46:29.607140+0000","last_fullsized":"2024-03-20T08:46:29.607140+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-20T08:45:52.630650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:45:52.630650+0000","last_clean_scrub_stamp":"2024-03-20T08:45:52.630650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:15:20.079183+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":29,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-20T08:46:29.606894+0000","last_change":"2024-03-20T08:45:49.631062+0000","last_active":"2024-03-20T08:46:29.606894+0000","last_peered":"2024-03-20T08:46:29.606894+0000","last_clean":"2024-03-20T08:46:29.606894+0000","last_became_active":"2024-03-20T08:45:49.630704+0000","last_became_peered":"2024-03-20T08:45:49.630704+0000","last_unstale":"2024-03-20T08:46:29.606894+0000","last_undegraded":"2024-03-20T08:46:29.606894+0000","last_fullsized":"2024-03-20T08:46:29.606894+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-20T08:45:48.603868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:45:48.603868+0000","last_clean_scrub_stamp":"2024-03-20T08:45:48.603868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T10:19:31.295642+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"67'9","reported_seq":41,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-20T08:46:29.658880+0000","last_change":"2024-03-20T08:45:51.646977+0000","last_active":"2024-03-20T08:46:29.658880+0000","last_peered":"2024-03-20T08:46:29.658880+0000","last_clean":"2024-03-20T08:46:29.658880+0000","last_became_active":"2024-03-20T08:45:51.646704+0000","last_became_peered":"2024-03-20T08:45:51.646704+0000","last_unstale":"2024-03-20T08:46:29.658880+0000","last_undegraded":"2024-03-20T08:46:29.658880+0000","last_fullsized":"2024-03-20T08:46:29.658880+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-20T08:45:50.616760+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:45:50.616760+0000","last_clean_scrub_stamp":"2024-03-20T08:45:50.616760+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T11:15:44.978821+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":17,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-20T08:46:29.606541+0000","last_change":"2024-03-20T08:45:55.689614+0000","last_active":"2024-03-20T08:46:29.606541+0000","last_peered":"2024-03-20T08:46:29.606541+0000","last_clean":"2024-03-20T08:46:29.606541+0000","last_became_active":"2024-03-20T08:45:55.689197+0000","last_became_peered":"2024-03-20T08:45:55.689197+0000","last_unstale":"2024-03-20T08:46:29.606541+0000","last_undegraded":"2024-03-20T08:46:29.606541+0000","last_fullsized":"2024-03-20T08:46:29.606541+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-20T08:45:54.654511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:45:54.654511+0000","last_clean_scrub_stamp":"2024-03-20T08:45:54.654511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T18:52:31.062423+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":21,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-20T08:46:29.683639+0000","last_change":"2024-03-20T08:45:53.664699+0000","last_active":"2024-03-20T08:46:29.683639+0000","last_peered":"2024-03-20T08:46:29.683639+0000","last_clean":"2024-03-20T08:46:29.683639+0000","last_became_active":"2024-03-20T08:45:53.663369+0000","last_became_peered":"2024-03-20T08:45:53.663369+0000","last_unstale":"2024-03-20T08:46:29.683639+0000","last_undegraded":"2024-03-20T08:46:29.683639+0000","last_fullsized":"2024-03-20T08:46:29.683639+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-20T08:45:52.630650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:45:52.630650+0000","last_clean_scrub_stamp":"2024-03-20T08:45:52.630650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T18:48:08.339895+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":29,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-20T08:46:29.607202+0000","last_change":"2024-03-20T08:45:49.627016+0000","last_active":"2024-03-20T08:46:29.607202+0000","last_peered":"2024-03-20T08:46:29.607202+0000","last_clean":"2024-03-20T08:46:29.607202+0000","last_became_active":"2024-03-20T08:45:49.626685+0000","last_became_peered":"2024-03-20T08:45:49.626685+0000","last_unstale":"2024-03-20T08:46:29.607202+0000","last_undegraded":"2024-03-20T08:46:29.607202+0000","last_fullsized":"2024-03-20T08:46:29.607202+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-20T08:45:48.603868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:45:48.603868+0000","last_clean_scrub_stamp":"2024-03-20T08:45:48.603868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T09:29:08.243782+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"67'5","reported_seq":35,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-20T08:46:29.658912+0000","last_change":"2024-03-20T08:45:51.652825+0000","last_active":"2024-03-20T08:46:29.658912+0000","last_peered":"2024-03-20T08:46:29.658912+0000","last_clean":"2024-03-20T08:46:29.658912+0000","last_became_active":"2024-03-20T08:45:51.652540+0000","last_became_peered":"2024-03-20T08:45:51.652540+0000","last_unstale":"2024-03-20T08:46:29.658912+0000","last_undegraded":"2024-03-20T08:46:29.658912+0000","last_fullsized":"2024-03-20T08:46:29.658912+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-20T08:45:50.616760+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:45:50.616760+0000","last_clean_scrub_stamp":"2024-03-20T08:45:50.616760+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T10:30:20.738617+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":7,"num_write":6,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":17,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-20T08:46:29.659315+0000","last_change":"2024-03-20T08:45:55.814816+0000","last_active":"2024-03-20T08:46:29.659315+0000","last_peered":"2024-03-20T08:46:29.659315+0000","last_clean":"2024-03-20T08:46:29.659315+0000","last_became_active":"2024-03-20T08:45:55.814511+0000","last_became_peered":"2024-03-20T08:45:55.814511+0000","last_unstale":"2024-03-20T08:46:29.659315+0000","last_undegraded":"2024-03-20T08:46:29.659315+0000","last_fullsized":"2024-03-20T08:46:29.659315+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-20T08:45:54.654511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:45:54.654511+0000","last_clean_scrub_stamp":"2024-03-20T08:45:54.654511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T11:57:19.915113+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":21,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-20T08:46:29.606987+0000","last_change":"2024-03-20T08:45:53.662681+0000","last_active":"2024-03-20T08:46:29.606987+0000","last_peered":"2024-03-20T08:46:29.606987+0000","last_clean":"2024-03-20T08:46:29.606987+0000","last_became_active":"2024-03-20T08:45:53.662060+0000","last_became_peered":"2024-03-20T08:45:53.662060+0000","last_unstale":"2024-03-20T08:46:29.606987+0000","last_undegraded":"2024-03-20T08:46:29.606987+0000","last_fullsized":"2024-03-20T08:46:29.606987+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-20T08:45:52.630650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:45:52.630650+0000","last_clean_scrub_stamp":"2024-03-20T08:45:52.630650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T17:04:40.358859+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":29,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-20T08:46:29.611069+0000","last_change":"2024-03-20T08:45:49.629580+0000","last_active":"2024-03-20T08:46:29.611069+0000","last_peered":"2024-03-20T08:46:29.611069+0000","last_clean":"2024-03-20T08:46:29.611069+0000","last_became_active":"2024-03-20T08:45:49.629271+0000","last_became_peered":"2024-03-20T08:45:49.629271+0000","last_unstale":"2024-03-20T08:46:29.611069+0000","last_undegraded":"2024-03-20T08:46:29.611069+0000","last_fullsized":"2024-03-20T08:46:29.611069+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-20T08:45:48.603868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:45:48.603868+0000","last_clean_scrub_stamp":"2024-03-20T08:45:48.603868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T20:16:47.586756+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"67'15","reported_seq":50,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-20T08:46:29.684270+0000","last_change":"2024-03-20T08:45:51.644824+0000","last_active":"2024-03-20T08:46:29.684270+0000","last_peered":"2024-03-20T08:46:29.684270+0000","last_clean":"2024-03-20T08:46:29.684270+0000","last_became_active":"2024-03-20T08:45:51.644533+0000","last_became_peered":"2024-03-20T08:45:51.644533+0000","last_unstale":"2024-03-20T08:46:29.684270+0000","last_undegraded":"2024-03-20T08:46:29.684270+0000","last_fullsized":"2024-03-20T08:46:29.684270+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-20T08:45:50.616760+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:45:50.616760+0000","last_clean_scrub_stamp":"2024-03-20T08:45:50.616760+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T12:56:51.465613+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":17,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-20T08:46:29.687264+0000","last_change":"2024-03-20T08:45:55.815272+0000","last_active":"2024-03-20T08:46:29.687264+0000","last_peered":"2024-03-20T08:46:29.687264+0000","last_clean":"2024-03-20T08:46:29.687264+0000","last_became_active":"2024-03-20T08:45:55.814811+0000","last_became_peered":"2024-03-20T08:45:55.814811+0000","last_unstale":"2024-03-20T08:46:29.687264+0000","last_undegraded":"2024-03-20T08:46:29.687264+0000","last_fullsized":"2024-03-20T08:46:29.687264+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-20T08:45:54.654511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:45:54.654511+0000","last_clean_scrub_stamp":"2024-03-20T08:45:54.654511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T15:07:10.829897+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":21,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-20T08:46:29.659062+0000","last_change":"2024-03-20T08:45:53.661844+0000","last_active":"2024-03-20T08:46:29.659062+0000","last_peered":"2024-03-20T08:46:29.659062+0000","last_clean":"2024-03-20T08:46:29.659062+0000","last_became_active":"2024-03-20T08:45:53.661546+0000","last_became_peered":"2024-03-20T08:45:53.661546+0000","last_unstale":"2024-03-20T08:46:29.659062+0000","last_undegraded":"2024-03-20T08:46:29.659062+0000","last_fullsized":"2024-03-20T08:46:29.659062+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-20T08:45:52.630650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:45:52.630650+0000","last_clean_scrub_stamp":"2024-03-20T08:45:52.630650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T18:20:17.789450+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":29,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-20T08:46:29.658281+0000","last_change":"2024-03-20T08:45:49.624287+0000","last_active":"2024-03-20T08:46:29.658281+0000","last_peered":"2024-03-20T08:46:29.658281+0000","last_clean":"2024-03-20T08:46:29.658281+0000","last_became_active":"2024-03-20T08:45:49.622798+0000","last_became_peered":"2024-03-20T08:45:49.622798+0000","last_unstale":"2024-03-20T08:46:29.658281+0000","last_undegraded":"2024-03-20T08:46:29.658281+0000","last_fullsized":"2024-03-20T08:46:29.658281+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-20T08:45:48.603868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:45:48.603868+0000","last_clean_scrub_stamp":"2024-03-20T08:45:48.603868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T11:45:14.127073+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"67'12","reported_seq":48,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-20T08:46:29.685645+0000","last_change":"2024-03-20T08:45:51.647664+0000","last_active":"2024-03-20T08:46:29.685645+0000","last_peered":"2024-03-20T08:46:29.685645+0000","last_clean":"2024-03-20T08:46:29.685645+0000","last_became_active":"2024-03-20T08:45:51.647382+0000","last_became_peered":"2024-03-20T08:45:51.647382+0000","last_unstale":"2024-03-20T08:46:29.685645+0000","last_undegraded":"2024-03-20T08:46:29.685645+0000","last_fullsized":"2024-03-20T08:46:29.685645+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-20T08:45:50.616760+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:45:50.616760+0000","last_clean_scrub_stamp":"2024-03-20T08:45:50.616760+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T19:49:37.729239+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1c","version":"67'1","reported_seq":19,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-20T08:46:29.683504+0000","last_change":"2024-03-20T08:45:55.689327+0000","last_active":"2024-03-20T08:46:29.683504+0000","last_peered":"2024-03-20T08:46:29.683504+0000","last_clean":"2024-03-20T08:46:29.683504+0000","last_became_active":"2024-03-20T08:45:55.689061+0000","last_became_peered":"2024-03-20T08:45:55.689061+0000","last_unstale":"2024-03-20T08:46:29.683504+0000","last_undegraded":"2024-03-20T08:46:29.683504+0000","last_fullsized":"2024-03-20T08:46:29.683504+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-20T08:45:54.654511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:45:54.654511+0000","last_clean_scrub_stamp":"2024-03-20T08:45:54.654511+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T15:10:26.339680+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":370,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1f","version":"68'11","reported_seq":54,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-20T08:47:14.687467+0000","last_change":"2024-03-20T08:45:53.657892+0000","last_active":"2024-03-20T08:47:14.687467+0000","last_peered":"2024-03-20T08:47:14.687467+0000","last_clean":"2024-03-20T08:47:14.687467+0000","last_became_active":"2024-03-20T08:45:53.657554+0000","last_became_peered":"2024-03-20T08:45:53.657554+0000","last_unstale":"2024-03-20T08:47:14.687467+0000","last_undegraded":"2024-03-20T08:47:14.687467+0000","last_fullsized":"2024-03-20T08:47:14.687467+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-20T08:45:52.630650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:45:52.630650+0000","last_clean_scrub_stamp":"2024-03-20T08:45:52.630650+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T18:28:18.424683+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.19","version":"60'1","reported_seq":30,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-20T08:46:29.605687+0000","last_change":"2024-03-20T08:45:49.629593+0000","last_active":"2024-03-20T08:46:29.605687+0000","last_peered":"2024-03-20T08:46:29.605687+0000","last_clean":"2024-03-20T08:46:29.605687+0000","last_became_active":"2024-03-20T08:45:49.629341+0000","last_became_peered":"2024-03-20T08:45:49.629341+0000","last_unstale":"2024-03-20T08:46:29.605687+0000","last_undegraded":"2024-03-20T08:46:29.605687+0000","last_fullsized":"2024-03-20T08:46:29.605687+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-20T08:45:48.603868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:45:48.603868+0000","last_clean_scrub_stamp":"2024-03-20T08:45:48.603868+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T09:46:55.930165+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"67'10","reported_seq":40,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-20T08:46:29.610410+0000","last_change":"2024-03-20T08:45:51.648496+0000","last_active":"2024-03-20T08:46:29.610410+0000","last_peered":"2024-03-20T08:46:29.610410+0000","last_clean":"2024-03-20T08:46:29.610410+0000","last_became_active":"2024-03-20T08:45:51.648200+0000","last_became_peered":"2024-03-20T08:45:51.648200+0000","last_unstale":"2024-03-20T08:46:29.610410+0000","last_undegraded":"2024-03-20T08:46:29.610410+0000","last_fullsized":"2024-03-20T08:46:29.610410+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-20T08:45:50.616760+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:45:50.616760+0000","last_clean_scrub_stamp":"2024-03-20T08:45:50.616760+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T17:16:49.051084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":17,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-20T08:46:29.606693+0000","last_change":"2024-03-20T08:45:55.685786+0000","last_active":"2024-03-20T08:46:29.606693+0000","last_peered":"2024-03-20T08:46:29.606693+0000","last_clean":"2024-03-20T08:46:29.606693+0000","last_became_active":"2024-03-20T08:45:55.685448+0000","last_became_peered":"2024-03-20T08:45:55.685448+0000","last_unstale":"2024-03-20T08:46:29.606693+0000","last_undegraded":"2024-03-20T08:46:29.606693+0000","last_fullsized":"2024-03-20T08:46:29.606693+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-20T08:45:54.654511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:45:54.654511+0000","last_clean_scrub_stamp":"2024-03-20T08:45:54.654511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T15:28:04.468079+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":21,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-20T08:46:29.610694+0000","last_change":"2024-03-20T08:45:53.661895+0000","last_active":"2024-03-20T08:46:29.610694+0000","last_peered":"2024-03-20T08:46:29.610694+0000","last_clean":"2024-03-20T08:46:29.610694+0000","last_became_active":"2024-03-20T08:45:53.661544+0000","last_became_peered":"2024-03-20T08:45:53.661544+0000","last_unstale":"2024-03-20T08:46:29.610694+0000","last_undegraded":"2024-03-20T08:46:29.610694+0000","last_fullsized":"2024-03-20T08:46:29.610694+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-20T08:45:52.630650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:45:52.630650+0000","last_clean_scrub_stamp":"2024-03-20T08:45:52.630650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T10:42:03.766549+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":29,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-20T08:46:29.685737+0000","last_change":"2024-03-20T08:45:49.631944+0000","last_active":"2024-03-20T08:46:29.685737+0000","last_peered":"2024-03-20T08:46:29.685737+0000","last_clean":"2024-03-20T08:46:29.685737+0000","last_became_active":"2024-03-20T08:45:49.631611+0000","last_became_peered":"2024-03-20T08:45:49.631611+0000","last_unstale":"2024-03-20T08:46:29.685737+0000","last_undegraded":"2024-03-20T08:46:29.685737+0000","last_fullsized":"2024-03-20T08:46:29.685737+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-20T08:45:48.603868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:45:48.603868+0000","last_clean_scrub_stamp":"2024-03-20T08:45:48.603868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:37:25.584469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"67'11","reported_seq":44,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-20T08:46:29.684557+0000","last_change":"2024-03-20T08:45:51.646441+0000","last_active":"2024-03-20T08:46:29.684557+0000","last_peered":"2024-03-20T08:46:29.684557+0000","last_clean":"2024-03-20T08:46:29.684557+0000","last_became_active":"2024-03-20T08:45:51.646163+0000","last_became_peered":"2024-03-20T08:45:51.646163+0000","last_unstale":"2024-03-20T08:46:29.684557+0000","last_undegraded":"2024-03-20T08:46:29.684557+0000","last_fullsized":"2024-03-20T08:46:29.684557+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-20T08:45:50.616760+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:45:50.616760+0000","last_clean_scrub_stamp":"2024-03-20T08:45:50.616760+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T19:44:40.076880+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":17,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-20T08:46:29.683534+0000","last_change":"2024-03-20T08:45:55.689249+0000","last_active":"2024-03-20T08:46:29.683534+0000","last_peered":"2024-03-20T08:46:29.683534+0000","last_clean":"2024-03-20T08:46:29.683534+0000","last_became_active":"2024-03-20T08:45:55.688961+0000","last_became_peered":"2024-03-20T08:45:55.688961+0000","last_unstale":"2024-03-20T08:46:29.683534+0000","last_undegraded":"2024-03-20T08:46:29.683534+0000","last_fullsized":"2024-03-20T08:46:29.683534+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-20T08:45:54.654511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:45:54.654511+0000","last_clean_scrub_stamp":"2024-03-20T08:45:54.654511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T16:39:22.227095+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":21,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-20T08:46:29.683311+0000","last_change":"2024-03-20T08:45:53.666010+0000","last_active":"2024-03-20T08:46:29.683311+0000","last_peered":"2024-03-20T08:46:29.683311+0000","last_clean":"2024-03-20T08:46:29.683311+0000","last_became_active":"2024-03-20T08:45:53.665686+0000","last_became_peered":"2024-03-20T08:45:53.665686+0000","last_unstale":"2024-03-20T08:46:29.683311+0000","last_undegraded":"2024-03-20T08:46:29.683311+0000","last_fullsized":"2024-03-20T08:46:29.683311+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-20T08:45:52.630650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:45:52.630650+0000","last_clean_scrub_stamp":"2024-03-20T08:45:52.630650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:04:15.415014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":29,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-20T08:46:29.611333+0000","last_change":"2024-03-20T08:45:49.627794+0000","last_active":"2024-03-20T08:46:29.611333+0000","last_peered":"2024-03-20T08:46:29.611333+0000","last_clean":"2024-03-20T08:46:29.611333+0000","last_became_active":"2024-03-20T08:45:49.627500+0000","last_became_peered":"2024-03-20T08:45:49.627500+0000","last_unstale":"2024-03-20T08:46:29.611333+0000","last_undegraded":"2024-03-20T08:46:29.611333+0000","last_fullsized":"2024-03-20T08:46:29.611333+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-20T08:45:48.603868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:45:48.603868+0000","last_clean_scrub_stamp":"2024-03-20T08:45:48.603868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T17:11:32.430384+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"67'4","reported_seq":31,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-20T08:46:29.685907+0000","last_change":"2024-03-20T08:45:51.646793+0000","last_active":"2024-03-20T08:46:29.685907+0000","last_peered":"2024-03-20T08:46:29.685907+0000","last_clean":"2024-03-20T08:46:29.685907+0000","last_became_active":"2024-03-20T08:45:51.646478+0000","last_became_peered":"2024-03-20T08:45:51.646478+0000","last_unstale":"2024-03-20T08:46:29.685907+0000","last_undegraded":"2024-03-20T08:46:29.685907+0000","last_fullsized":"2024-03-20T08:46:29.685907+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-20T08:45:50.616760+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:45:50.616760+0000","last_clean_scrub_stamp":"2024-03-20T08:45:50.616760+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T16:26:49.832509+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":17,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-20T08:46:29.686665+0000","last_change":"2024-03-20T08:45:55.815899+0000","last_active":"2024-03-20T08:46:29.686665+0000","last_peered":"2024-03-20T08:46:29.686665+0000","last_clean":"2024-03-20T08:46:29.686665+0000","last_became_active":"2024-03-20T08:45:55.815317+0000","last_became_peered":"2024-03-20T08:45:55.815317+0000","last_unstale":"2024-03-20T08:46:29.686665+0000","last_undegraded":"2024-03-20T08:46:29.686665+0000","last_fullsized":"2024-03-20T08:46:29.686665+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-20T08:45:54.654511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:45:54.654511+0000","last_clean_scrub_stamp":"2024-03-20T08:45:54.654511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:13:59.756720+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":21,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-20T08:46:29.684413+0000","last_change":"2024-03-20T08:45:53.663516+0000","last_active":"2024-03-20T08:46:29.684413+0000","last_peered":"2024-03-20T08:46:29.684413+0000","last_clean":"2024-03-20T08:46:29.684413+0000","last_became_active":"2024-03-20T08:45:53.663221+0000","last_became_peered":"2024-03-20T08:45:53.663221+0000","last_unstale":"2024-03-20T08:46:29.684413+0000","last_undegraded":"2024-03-20T08:46:29.684413+0000","last_fullsized":"2024-03-20T08:46:29.684413+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-20T08:45:52.630650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:45:52.630650+0000","last_clean_scrub_stamp":"2024-03-20T08:45:52.630650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T16:46:27.073017+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.16","version":"60'1","reported_seq":37,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-20T08:46:29.607523+0000","last_change":"2024-03-20T08:45:49.633276+0000","last_active":"2024-03-20T08:46:29.607523+0000","last_peered":"2024-03-20T08:46:29.607523+0000","last_clean":"2024-03-20T08:46:29.607523+0000","last_became_active":"2024-03-20T08:45:49.632704+0000","last_became_peered":"2024-03-20T08:45:49.632704+0000","last_unstale":"2024-03-20T08:46:29.607523+0000","last_undegraded":"2024-03-20T08:46:29.607523+0000","last_fullsized":"2024-03-20T08:46:29.607523+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-20T08:45:48.603868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:45:48.603868+0000","last_clean_scrub_stamp":"2024-03-20T08:45:48.603868+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T10:39:25.298178+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":911,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.11","version":"67'11","reported_seq":44,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-20T08:46:29.685082+0000","last_change":"2024-03-20T08:45:51.650889+0000","last_active":"2024-03-20T08:46:29.685082+0000","last_peered":"2024-03-20T08:46:29.685082+0000","last_clean":"2024-03-20T08:46:29.685082+0000","last_became_active":"2024-03-20T08:45:51.650596+0000","last_became_peered":"2024-03-20T08:45:51.650596+0000","last_unstale":"2024-03-20T08:46:29.685082+0000","last_undegraded":"2024-03-20T08:46:29.685082+0000","last_fullsized":"2024-03-20T08:46:29.685082+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-20T08:45:50.616760+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:45:50.616760+0000","last_clean_scrub_stamp":"2024-03-20T08:45:50.616760+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T11:22:00.070168+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":17,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-20T08:46:29.609700+0000","last_change":"2024-03-20T08:45:55.687298+0000","last_active":"2024-03-20T08:46:29.609700+0000","last_peered":"2024-03-20T08:46:29.609700+0000","last_clean":"2024-03-20T08:46:29.609700+0000","last_became_active":"2024-03-20T08:45:55.686910+0000","last_became_peered":"2024-03-20T08:45:55.686910+0000","last_unstale":"2024-03-20T08:46:29.609700+0000","last_undegraded":"2024-03-20T08:46:29.609700+0000","last_fullsized":"2024-03-20T08:46:29.609700+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-20T08:45:54.654511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:45:54.654511+0000","last_clean_scrub_stamp":"2024-03-20T08:45:54.654511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T11:31:36.817467+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":21,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-20T08:46:29.686938+0000","last_change":"2024-03-20T08:45:53.655655+0000","last_active":"2024-03-20T08:46:29.686938+0000","last_peered":"2024-03-20T08:46:29.686938+0000","last_clean":"2024-03-20T08:46:29.686938+0000","last_became_active":"2024-03-20T08:45:53.655269+0000","last_became_peered":"2024-03-20T08:45:53.655269+0000","last_unstale":"2024-03-20T08:46:29.686938+0000","last_undegraded":"2024-03-20T08:46:29.686938+0000","last_fullsized":"2024-03-20T08:46:29.686938+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-20T08:45:52.630650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:45:52.630650+0000","last_clean_scrub_stamp":"2024-03-20T08:45:52.630650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:07:06.161785+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.15","version":"60'1","reported_seq":37,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-20T08:46:29.684666+0000","last_change":"2024-03-20T08:45:49.629961+0000","last_active":"2024-03-20T08:46:29.684666+0000","last_peered":"2024-03-20T08:46:29.684666+0000","last_clean":"2024-03-20T08:46:29.684666+0000","last_became_active":"2024-03-20T08:45:49.629491+0000","last_became_peered":"2024-03-20T08:45:49.629491+0000","last_unstale":"2024-03-20T08:46:29.684666+0000","last_undegraded":"2024-03-20T08:46:29.684666+0000","last_fullsized":"2024-03-20T08:46:29.684666+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-20T08:45:48.603868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:45:48.603868+0000","last_clean_scrub_stamp":"2024-03-20T08:45:48.603868+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T17:07:22.223272+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":436,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.12","version":"67'9","reported_seq":41,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-20T08:46:29.606473+0000","last_change":"2024-03-20T08:45:51.642899+0000","last_active":"2024-03-20T08:46:29.606473+0000","last_peered":"2024-03-20T08:46:29.606473+0000","last_clean":"2024-03-20T08:46:29.606473+0000","last_became_active":"2024-03-20T08:45:51.642610+0000","last_became_peered":"2024-03-20T08:45:51.642610+0000","last_unstale":"2024-03-20T08:46:29.606473+0000","last_undegraded":"2024-03-20T08:46:29.606473+0000","last_fullsized":"2024-03-20T08:46:29.606473+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-20T08:45:50.616760+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:45:50.616760+0000","last_clean_scrub_stamp":"2024-03-20T08:45:50.616760+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T12:24:22.517141+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":17,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-20T08:46:29.684780+0000","last_change":"2024-03-20T08:45:55.692907+0000","last_active":"2024-03-20T08:46:29.684780+0000","last_peered":"2024-03-20T08:46:29.684780+0000","last_clean":"2024-03-20T08:46:29.684780+0000","last_became_active":"2024-03-20T08:45:55.691558+0000","last_became_peered":"2024-03-20T08:45:55.691558+0000","last_unstale":"2024-03-20T08:46:29.684780+0000","last_undegraded":"2024-03-20T08:46:29.684780+0000","last_fullsized":"2024-03-20T08:46:29.684780+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-20T08:45:54.654511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:45:54.654511+0000","last_clean_scrub_stamp":"2024-03-20T08:45:54.654511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T19:33:31.509549+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":21,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-20T08:46:29.606246+0000","last_change":"2024-03-20T08:45:53.657596+0000","last_active":"2024-03-20T08:46:29.606246+0000","last_peered":"2024-03-20T08:46:29.606246+0000","last_clean":"2024-03-20T08:46:29.606246+0000","last_became_active":"2024-03-20T08:45:53.656954+0000","last_became_peered":"2024-03-20T08:45:53.656954+0000","last_unstale":"2024-03-20T08:46:29.606246+0000","last_undegraded":"2024-03-20T08:46:29.606246+0000","last_fullsized":"2024-03-20T08:46:29.606246+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-20T08:45:52.630650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:45:52.630650+0000","last_clean_scrub_stamp":"2024-03-20T08:45:52.630650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T10:31:30.378735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":29,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-20T08:46:29.659254+0000","last_change":"2024-03-20T08:45:49.629013+0000","last_active":"2024-03-20T08:46:29.659254+0000","last_peered":"2024-03-20T08:46:29.659254+0000","last_clean":"2024-03-20T08:46:29.659254+0000","last_became_active":"2024-03-20T08:45:49.628287+0000","last_became_peered":"2024-03-20T08:45:49.628287+0000","last_unstale":"2024-03-20T08:46:29.659254+0000","last_undegraded":"2024-03-20T08:46:29.659254+0000","last_fullsized":"2024-03-20T08:46:29.659254+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-20T08:45:48.603868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:45:48.603868+0000","last_clean_scrub_stamp":"2024-03-20T08:45:48.603868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T17:14:24.813516+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"67'11","reported_seq":44,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-20T08:46:29.659050+0000","last_change":"2024-03-20T08:45:51.652783+0000","last_active":"2024-03-20T08:46:29.659050+0000","last_peered":"2024-03-20T08:46:29.659050+0000","last_clean":"2024-03-20T08:46:29.659050+0000","last_became_active":"2024-03-20T08:45:51.652485+0000","last_became_peered":"2024-03-20T08:45:51.652485+0000","last_unstale":"2024-03-20T08:46:29.659050+0000","last_undegraded":"2024-03-20T08:46:29.659050+0000","last_fullsized":"2024-03-20T08:46:29.659050+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-20T08:45:50.616760+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:45:50.616760+0000","last_clean_scrub_stamp":"2024-03-20T08:45:50.616760+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:22:17.120553+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":17,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-20T08:46:29.610029+0000","last_change":"2024-03-20T08:45:55.693809+0000","last_active":"2024-03-20T08:46:29.610029+0000","last_peered":"2024-03-20T08:46:29.610029+0000","last_clean":"2024-03-20T08:46:29.610029+0000","last_became_active":"2024-03-20T08:45:55.693482+0000","last_became_peered":"2024-03-20T08:45:55.693482+0000","last_unstale":"2024-03-20T08:46:29.610029+0000","last_undegraded":"2024-03-20T08:46:29.610029+0000","last_fullsized":"2024-03-20T08:46:29.610029+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-20T08:45:54.654511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:45:54.654511+0000","last_clean_scrub_stamp":"2024-03-20T08:45:54.654511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T15:59:51.338868+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.15","version":"68'11","reported_seq":51,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-20T08:47:14.687472+0000","last_change":"2024-03-20T08:45:53.663741+0000","last_active":"2024-03-20T08:47:14.687472+0000","last_peered":"2024-03-20T08:47:14.687472+0000","last_clean":"2024-03-20T08:47:14.687472+0000","last_became_active":"2024-03-20T08:45:53.663452+0000","last_became_peered":"2024-03-20T08:45:53.663452+0000","last_unstale":"2024-03-20T08:47:14.687472+0000","last_undegraded":"2024-03-20T08:47:14.687472+0000","last_fullsized":"2024-03-20T08:47:14.687472+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-20T08:45:52.630650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:45:52.630650+0000","last_clean_scrub_stamp":"2024-03-20T08:45:52.630650+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T16:42:16.218775+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":29,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-20T08:46:29.683850+0000","last_change":"2024-03-20T08:45:49.631507+0000","last_active":"2024-03-20T08:46:29.683850+0000","last_peered":"2024-03-20T08:46:29.683850+0000","last_clean":"2024-03-20T08:46:29.683850+0000","last_became_active":"2024-03-20T08:45:49.631193+0000","last_became_peered":"2024-03-20T08:45:49.631193+0000","last_unstale":"2024-03-20T08:46:29.683850+0000","last_undegraded":"2024-03-20T08:46:29.683850+0000","last_fullsized":"2024-03-20T08:46:29.683850+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-20T08:45:48.603868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:45:48.603868+0000","last_clean_scrub_stamp":"2024-03-20T08:45:48.603868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T18:58:24.656178+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"67'10","reported_seq":40,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-20T08:46:29.687033+0000","last_change":"2024-03-20T08:45:51.649967+0000","last_active":"2024-03-20T08:46:29.687033+0000","last_peered":"2024-03-20T08:46:29.687033+0000","last_clean":"2024-03-20T08:46:29.687033+0000","last_became_active":"2024-03-20T08:45:51.649657+0000","last_became_peered":"2024-03-20T08:45:51.649657+0000","last_unstale":"2024-03-20T08:46:29.687033+0000","last_undegraded":"2024-03-20T08:46:29.687033+0000","last_fullsized":"2024-03-20T08:46:29.687033+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-20T08:45:50.616760+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:45:50.616760+0000","last_clean_scrub_stamp":"2024-03-20T08:45:50.616760+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T19:34:55.533460+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":17,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-20T08:46:29.659612+0000","last_change":"2024-03-20T08:45:55.687183+0000","last_active":"2024-03-20T08:46:29.659612+0000","last_peered":"2024-03-20T08:46:29.659612+0000","last_clean":"2024-03-20T08:46:29.659612+0000","last_became_active":"2024-03-20T08:45:55.686757+0000","last_became_peered":"2024-03-20T08:45:55.686757+0000","last_unstale":"2024-03-20T08:46:29.659612+0000","last_undegraded":"2024-03-20T08:46:29.659612+0000","last_fullsized":"2024-03-20T08:46:29.659612+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-20T08:45:54.654511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:45:54.654511+0000","last_clean_scrub_stamp":"2024-03-20T08:45:54.654511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:14:14.232613+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.14","version":"68'11","reported_seq":51,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-20T08:47:14.687579+0000","last_change":"2024-03-20T08:45:53.665034+0000","last_active":"2024-03-20T08:47:14.687579+0000","last_peered":"2024-03-20T08:47:14.687579+0000","last_clean":"2024-03-20T08:47:14.687579+0000","last_became_active":"2024-03-20T08:45:53.664661+0000","last_became_peered":"2024-03-20T08:45:53.664661+0000","last_unstale":"2024-03-20T08:47:14.687579+0000","last_undegraded":"2024-03-20T08:47:14.687579+0000","last_fullsized":"2024-03-20T08:47:14.687579+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-20T08:45:52.630650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:45:52.630650+0000","last_clean_scrub_stamp":"2024-03-20T08:45:52.630650+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T19:43:34.419422+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":29,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-20T08:46:29.610622+0000","last_change":"2024-03-20T08:45:49.628884+0000","last_active":"2024-03-20T08:46:29.610622+0000","last_peered":"2024-03-20T08:46:29.610622+0000","last_clean":"2024-03-20T08:46:29.610622+0000","last_became_active":"2024-03-20T08:45:49.628553+0000","last_became_peered":"2024-03-20T08:45:49.628553+0000","last_unstale":"2024-03-20T08:46:29.610622+0000","last_undegraded":"2024-03-20T08:46:29.610622+0000","last_fullsized":"2024-03-20T08:46:29.610622+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-20T08:45:48.603868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:45:48.603868+0000","last_clean_scrub_stamp":"2024-03-20T08:45:48.603868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T12:59:19.958241+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"67'9","reported_seq":41,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-20T08:46:29.607591+0000","last_change":"2024-03-20T08:45:51.645538+0000","last_active":"2024-03-20T08:46:29.607591+0000","last_peered":"2024-03-20T08:46:29.607591+0000","last_clean":"2024-03-20T08:46:29.607591+0000","last_became_active":"2024-03-20T08:45:51.644458+0000","last_became_peered":"2024-03-20T08:45:51.644458+0000","last_unstale":"2024-03-20T08:46:29.607591+0000","last_undegraded":"2024-03-20T08:46:29.607591+0000","last_fullsized":"2024-03-20T08:46:29.607591+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-20T08:45:50.616760+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:45:50.616760+0000","last_clean_scrub_stamp":"2024-03-20T08:45:50.616760+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T18:21:22.224425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":17,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-20T08:46:29.684969+0000","last_change":"2024-03-20T08:45:55.689296+0000","last_active":"2024-03-20T08:46:29.684969+0000","last_peered":"2024-03-20T08:46:29.684969+0000","last_clean":"2024-03-20T08:46:29.684969+0000","last_became_active":"2024-03-20T08:45:55.688994+0000","last_became_peered":"2024-03-20T08:45:55.688994+0000","last_unstale":"2024-03-20T08:46:29.684969+0000","last_undegraded":"2024-03-20T08:46:29.684969+0000","last_fullsized":"2024-03-20T08:46:29.684969+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-20T08:45:54.654511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:45:54.654511+0000","last_clean_scrub_stamp":"2024-03-20T08:45:54.654511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T11:06:35.965720+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":21,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-20T08:46:29.687582+0000","last_change":"2024-03-20T08:45:53.660837+0000","last_active":"2024-03-20T08:46:29.687582+0000","last_peered":"2024-03-20T08:46:29.687582+0000","last_clean":"2024-03-20T08:46:29.687582+0000","last_became_active":"2024-03-20T08:45:53.660512+0000","last_became_peered":"2024-03-20T08:45:53.660512+0000","last_unstale":"2024-03-20T08:46:29.687582+0000","last_undegraded":"2024-03-20T08:46:29.687582+0000","last_fullsized":"2024-03-20T08:46:29.687582+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-20T08:45:52.630650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:45:52.630650+0000","last_clean_scrub_stamp":"2024-03-20T08:45:52.630650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T17:46:25.157304+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":29,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-20T08:46:29.684066+0000","last_change":"2024-03-20T08:45:49.635863+0000","last_active":"2024-03-20T08:46:29.684066+0000","last_peered":"2024-03-20T08:46:29.684066+0000","last_clean":"2024-03-20T08:46:29.684066+0000","last_became_active":"2024-03-20T08:45:49.635570+0000","last_became_peered":"2024-03-20T08:45:49.635570+0000","last_unstale":"2024-03-20T08:46:29.684066+0000","last_undegraded":"2024-03-20T08:46:29.684066+0000","last_fullsized":"2024-03-20T08:46:29.684066+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-20T08:45:48.603868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:45:48.603868+0000","last_clean_scrub_stamp":"2024-03-20T08:45:48.603868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T09:38:45.343126+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"67'9","reported_seq":41,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-20T08:46:29.610525+0000","last_change":"2024-03-20T08:45:51.648332+0000","last_active":"2024-03-20T08:46:29.610525+0000","last_peered":"2024-03-20T08:46:29.610525+0000","last_clean":"2024-03-20T08:46:29.610525+0000","last_became_active":"2024-03-20T08:45:51.648062+0000","last_became_peered":"2024-03-20T08:45:51.648062+0000","last_unstale":"2024-03-20T08:46:29.610525+0000","last_undegraded":"2024-03-20T08:46:29.610525+0000","last_fullsized":"2024-03-20T08:46:29.610525+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-20T08:45:50.616760+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:45:50.616760+0000","last_clean_scrub_stamp":"2024-03-20T08:45:50.616760+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T16:39:51.402299+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":17,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-20T08:46:29.684579+0000","last_change":"2024-03-20T08:45:55.816865+0000","last_active":"2024-03-20T08:46:29.684579+0000","last_peered":"2024-03-20T08:46:29.684579+0000","last_clean":"2024-03-20T08:46:29.684579+0000","last_became_active":"2024-03-20T08:45:55.816554+0000","last_became_peered":"2024-03-20T08:45:55.816554+0000","last_unstale":"2024-03-20T08:46:29.684579+0000","last_undegraded":"2024-03-20T08:46:29.684579+0000","last_fullsized":"2024-03-20T08:46:29.684579+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-20T08:45:54.654511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:45:54.654511+0000","last_clean_scrub_stamp":"2024-03-20T08:45:54.654511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:27:53.618904+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":21,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-20T08:46:29.607637+0000","last_change":"2024-03-20T08:45:53.660167+0000","last_active":"2024-03-20T08:46:29.607637+0000","last_peered":"2024-03-20T08:46:29.607637+0000","last_clean":"2024-03-20T08:46:29.607637+0000","last_became_active":"2024-03-20T08:45:53.659874+0000","last_became_peered":"2024-03-20T08:45:53.659874+0000","last_unstale":"2024-03-20T08:46:29.607637+0000","last_undegraded":"2024-03-20T08:46:29.607637+0000","last_fullsized":"2024-03-20T08:46:29.607637+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-20T08:45:52.630650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:45:52.630650+0000","last_clean_scrub_stamp":"2024-03-20T08:45:52.630650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T10:17:48.470760+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":29,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-20T08:46:29.683184+0000","last_change":"2024-03-20T08:45:49.627698+0000","last_active":"2024-03-20T08:46:29.683184+0000","last_peered":"2024-03-20T08:46:29.683184+0000","last_clean":"2024-03-20T08:46:29.683184+0000","last_became_active":"2024-03-20T08:45:49.627468+0000","last_became_peered":"2024-03-20T08:45:49.627468+0000","last_unstale":"2024-03-20T08:46:29.683184+0000","last_undegraded":"2024-03-20T08:46:29.683184+0000","last_fullsized":"2024-03-20T08:46:29.683184+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-20T08:45:48.603868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:45:48.603868+0000","last_clean_scrub_stamp":"2024-03-20T08:45:48.603868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T18:00:26.933257+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"67'6","reported_seq":34,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-20T08:46:29.687463+0000","last_change":"2024-03-20T08:45:51.650932+0000","last_active":"2024-03-20T08:46:29.687463+0000","last_peered":"2024-03-20T08:46:29.687463+0000","last_clean":"2024-03-20T08:46:29.687463+0000","last_became_active":"2024-03-20T08:45:51.650635+0000","last_became_peered":"2024-03-20T08:45:51.650635+0000","last_unstale":"2024-03-20T08:46:29.687463+0000","last_undegraded":"2024-03-20T08:46:29.687463+0000","last_fullsized":"2024-03-20T08:46:29.687463+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-20T08:45:50.616760+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:45:50.616760+0000","last_clean_scrub_stamp":"2024-03-20T08:45:50.616760+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T09:20:24.448441+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":9,"num_read_kb":6,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":17,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-20T08:46:29.607106+0000","last_change":"2024-03-20T08:45:55.814872+0000","last_active":"2024-03-20T08:46:29.607106+0000","last_peered":"2024-03-20T08:46:29.607106+0000","last_clean":"2024-03-20T08:46:29.607106+0000","last_became_active":"2024-03-20T08:45:55.814481+0000","last_became_peered":"2024-03-20T08:45:55.814481+0000","last_unstale":"2024-03-20T08:46:29.607106+0000","last_undegraded":"2024-03-20T08:46:29.607106+0000","last_fullsized":"2024-03-20T08:46:29.607106+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-20T08:45:54.654511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:45:54.654511+0000","last_clean_scrub_stamp":"2024-03-20T08:45:54.654511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T12:49:29.493130+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"60'2","reported_seq":45,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-20T08:46:29.683555+0000","last_change":"2024-03-20T08:45:49.634369+0000","last_active":"2024-03-20T08:46:29.683555+0000","last_peered":"2024-03-20T08:46:29.683555+0000","last_clean":"2024-03-20T08:46:29.683555+0000","last_became_active":"2024-03-20T08:45:49.633882+0000","last_became_peered":"2024-03-20T08:45:49.633882+0000","last_unstale":"2024-03-20T08:46:29.683555+0000","last_undegraded":"2024-03-20T08:46:29.683555+0000","last_fullsized":"2024-03-20T08:46:29.683555+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-20T08:45:48.603868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:45:48.603868+0000","last_clean_scrub_stamp":"2024-03-20T08:45:48.603868+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T11:22:20.528408+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":14,"num_read_kb":14,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"67'15","reported_seq":50,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-20T08:46:29.606618+0000","last_change":"2024-03-20T08:45:51.641571+0000","last_active":"2024-03-20T08:46:29.606618+0000","last_peered":"2024-03-20T08:46:29.606618+0000","last_clean":"2024-03-20T08:46:29.606618+0000","last_became_active":"2024-03-20T08:45:51.641144+0000","last_became_peered":"2024-03-20T08:45:51.641144+0000","last_unstale":"2024-03-20T08:46:29.606618+0000","last_undegraded":"2024-03-20T08:46:29.606618+0000","last_fullsized":"2024-03-20T08:46:29.606618+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-20T08:45:50.616760+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:45:50.616760+0000","last_clean_scrub_stamp":"2024-03-20T08:45:50.616760+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T16:35:21.951940+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"68'11","reported_seq":51,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-20T08:47:14.687512+0000","last_change":"2024-03-20T08:45:53.665167+0000","last_active":"2024-03-20T08:47:14.687512+0000","last_peered":"2024-03-20T08:47:14.687512+0000","last_clean":"2024-03-20T08:47:14.687512+0000","last_became_active":"2024-03-20T08:45:53.664813+0000","last_became_peered":"2024-03-20T08:45:53.664813+0000","last_unstale":"2024-03-20T08:47:14.687512+0000","last_undegraded":"2024-03-20T08:47:14.687512+0000","last_fullsized":"2024-03-20T08:47:14.687512+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-20T08:45:52.630650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:45:52.630650+0000","last_clean_scrub_stamp":"2024-03-20T08:45:52.630650+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T11:17:18.224664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":17,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-20T08:46:29.686756+0000","last_change":"2024-03-20T08:45:55.693636+0000","last_active":"2024-03-20T08:46:29.686756+0000","last_peered":"2024-03-20T08:46:29.686756+0000","last_clean":"2024-03-20T08:46:29.686756+0000","last_became_active":"2024-03-20T08:45:55.693321+0000","last_became_peered":"2024-03-20T08:45:55.693321+0000","last_unstale":"2024-03-20T08:46:29.686756+0000","last_undegraded":"2024-03-20T08:46:29.686756+0000","last_fullsized":"2024-03-20T08:46:29.686756+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-20T08:45:54.654511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:45:54.654511+0000","last_clean_scrub_stamp":"2024-03-20T08:45:54.654511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T18:35:56.615151+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":29,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-20T08:46:29.683788+0000","last_change":"2024-03-20T08:45:49.631139+0000","last_active":"2024-03-20T08:46:29.683788+0000","last_peered":"2024-03-20T08:46:29.683788+0000","last_clean":"2024-03-20T08:46:29.683788+0000","last_became_active":"2024-03-20T08:45:49.630850+0000","last_became_peered":"2024-03-20T08:45:49.630850+0000","last_unstale":"2024-03-20T08:46:29.683788+0000","last_undegraded":"2024-03-20T08:46:29.683788+0000","last_fullsized":"2024-03-20T08:46:29.683788+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-20T08:45:48.603868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:45:48.603868+0000","last_clean_scrub_stamp":"2024-03-20T08:45:48.603868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T18:23:48.781961+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"67'12","reported_seq":48,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-20T08:46:29.658895+0000","last_change":"2024-03-20T08:45:51.648425+0000","last_active":"2024-03-20T08:46:29.658895+0000","last_peered":"2024-03-20T08:46:29.658895+0000","last_clean":"2024-03-20T08:46:29.658895+0000","last_became_active":"2024-03-20T08:45:51.648094+0000","last_became_peered":"2024-03-20T08:45:51.648094+0000","last_unstale":"2024-03-20T08:46:29.658895+0000","last_undegraded":"2024-03-20T08:46:29.658895+0000","last_fullsized":"2024-03-20T08:46:29.658895+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-20T08:45:50.616760+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:45:50.616760+0000","last_clean_scrub_stamp":"2024-03-20T08:45:50.616760+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T20:24:59.105925+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":21,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-20T08:46:29.684269+0000","last_change":"2024-03-20T08:45:53.667088+0000","last_active":"2024-03-20T08:46:29.684269+0000","last_peered":"2024-03-20T08:46:29.684269+0000","last_clean":"2024-03-20T08:46:29.684269+0000","last_became_active":"2024-03-20T08:45:53.666769+0000","last_became_peered":"2024-03-20T08:45:53.666769+0000","last_unstale":"2024-03-20T08:46:29.684269+0000","last_undegraded":"2024-03-20T08:46:29.684269+0000","last_fullsized":"2024-03-20T08:46:29.684269+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-20T08:45:52.630650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:45:52.630650+0000","last_clean_scrub_stamp":"2024-03-20T08:45:52.630650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:12:02.183265+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":17,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-20T08:46:29.683649+0000","last_change":"2024-03-20T08:45:55.693463+0000","last_active":"2024-03-20T08:46:29.683649+0000","last_peered":"2024-03-20T08:46:29.683649+0000","last_clean":"2024-03-20T08:46:29.683649+0000","last_became_active":"2024-03-20T08:45:55.693220+0000","last_became_peered":"2024-03-20T08:45:55.693220+0000","last_unstale":"2024-03-20T08:46:29.683649+0000","last_undegraded":"2024-03-20T08:46:29.683649+0000","last_fullsized":"2024-03-20T08:46:29.683649+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-20T08:45:54.654511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:45:54.654511+0000","last_clean_scrub_stamp":"2024-03-20T08:45:54.654511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:15:44.654763+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":21,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-20T08:46:29.684225+0000","last_change":"2024-03-20T08:45:53.667191+0000","last_active":"2024-03-20T08:46:29.684225+0000","last_peered":"2024-03-20T08:46:29.684225+0000","last_clean":"2024-03-20T08:46:29.684225+0000","last_became_active":"2024-03-20T08:45:53.666883+0000","last_became_peered":"2024-03-20T08:45:53.666883+0000","last_unstale":"2024-03-20T08:46:29.684225+0000","last_undegraded":"2024-03-20T08:46:29.684225+0000","last_fullsized":"2024-03-20T08:46:29.684225+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-20T08:45:52.630650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:45:52.630650+0000","last_clean_scrub_stamp":"2024-03-20T08:45:52.630650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T15:06:50.829308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":29,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-20T08:46:29.683217+0000","last_change":"2024-03-20T08:45:49.635051+0000","last_active":"2024-03-20T08:46:29.683217+0000","last_peered":"2024-03-20T08:46:29.683217+0000","last_clean":"2024-03-20T08:46:29.683217+0000","last_became_active":"2024-03-20T08:45:49.634085+0000","last_became_peered":"2024-03-20T08:45:49.634085+0000","last_unstale":"2024-03-20T08:46:29.683217+0000","last_undegraded":"2024-03-20T08:46:29.683217+0000","last_fullsized":"2024-03-20T08:46:29.683217+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-20T08:45:48.603868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:45:48.603868+0000","last_clean_scrub_stamp":"2024-03-20T08:45:48.603868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T16:36:22.957495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"67'19","reported_seq":56,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-20T08:46:29.684043+0000","last_change":"2024-03-20T08:45:51.646441+0000","last_active":"2024-03-20T08:46:29.684043+0000","last_peered":"2024-03-20T08:46:29.684043+0000","last_clean":"2024-03-20T08:46:29.684043+0000","last_became_active":"2024-03-20T08:45:51.646126+0000","last_became_peered":"2024-03-20T08:45:51.646126+0000","last_unstale":"2024-03-20T08:46:29.684043+0000","last_undegraded":"2024-03-20T08:46:29.684043+0000","last_fullsized":"2024-03-20T08:46:29.684043+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-20T08:45:50.616760+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:45:50.616760+0000","last_clean_scrub_stamp":"2024-03-20T08:45:50.616760+0000","objects_scrubbed":0,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T09:59:33.305144+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":32,"num_read_kb":21,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":17,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-20T08:46:29.610157+0000","last_change":"2024-03-20T08:45:55.687980+0000","last_active":"2024-03-20T08:46:29.610157+0000","last_peered":"2024-03-20T08:46:29.610157+0000","last_clean":"2024-03-20T08:46:29.610157+0000","last_became_active":"2024-03-20T08:45:55.687230+0000","last_became_peered":"2024-03-20T08:45:55.687230+0000","last_unstale":"2024-03-20T08:46:29.610157+0000","last_undegraded":"2024-03-20T08:46:29.610157+0000","last_fullsized":"2024-03-20T08:46:29.610157+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-20T08:45:54.654511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:45:54.654511+0000","last_clean_scrub_stamp":"2024-03-20T08:45:54.654511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T09:33:04.045483+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":21,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-20T08:46:29.684326+0000","last_change":"2024-03-20T08:45:53.665316+0000","last_active":"2024-03-20T08:46:29.684326+0000","last_peered":"2024-03-20T08:46:29.684326+0000","last_clean":"2024-03-20T08:46:29.684326+0000","last_became_active":"2024-03-20T08:45:53.664079+0000","last_became_peered":"2024-03-20T08:45:53.664079+0000","last_unstale":"2024-03-20T08:46:29.684326+0000","last_undegraded":"2024-03-20T08:46:29.684326+0000","last_fullsized":"2024-03-20T08:46:29.684326+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-20T08:45:52.630650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:45:52.630650+0000","last_clean_scrub_stamp":"2024-03-20T08:45:52.630650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:28:50.692891+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":29,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-20T08:46:29.607263+0000","last_change":"2024-03-20T08:45:49.630947+0000","last_active":"2024-03-20T08:46:29.607263+0000","last_peered":"2024-03-20T08:46:29.607263+0000","last_clean":"2024-03-20T08:46:29.607263+0000","last_became_active":"2024-03-20T08:45:49.630606+0000","last_became_peered":"2024-03-20T08:45:49.630606+0000","last_unstale":"2024-03-20T08:46:29.607263+0000","last_undegraded":"2024-03-20T08:46:29.607263+0000","last_fullsized":"2024-03-20T08:46:29.607263+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-20T08:45:48.603868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:45:48.603868+0000","last_clean_scrub_stamp":"2024-03-20T08:45:48.603868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T17:21:13.198286+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"67'9","reported_seq":41,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-20T08:46:29.611339+0000","last_change":"2024-03-20T08:45:51.643356+0000","last_active":"2024-03-20T08:46:29.611339+0000","last_peered":"2024-03-20T08:46:29.611339+0000","last_clean":"2024-03-20T08:46:29.611339+0000","last_became_active":"2024-03-20T08:45:51.643069+0000","last_became_peered":"2024-03-20T08:45:51.643069+0000","last_unstale":"2024-03-20T08:46:29.611339+0000","last_undegraded":"2024-03-20T08:46:29.611339+0000","last_fullsized":"2024-03-20T08:46:29.611339+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-20T08:45:50.616760+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:45:50.616760+0000","last_clean_scrub_stamp":"2024-03-20T08:45:50.616760+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T19:30:50.784255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":17,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-20T08:46:29.684553+0000","last_change":"2024-03-20T08:45:55.694077+0000","last_active":"2024-03-20T08:46:29.684553+0000","last_peered":"2024-03-20T08:46:29.684553+0000","last_clean":"2024-03-20T08:46:29.684553+0000","last_became_active":"2024-03-20T08:45:55.693647+0000","last_became_peered":"2024-03-20T08:45:55.693647+0000","last_unstale":"2024-03-20T08:46:29.684553+0000","last_undegraded":"2024-03-20T08:46:29.684553+0000","last_fullsized":"2024-03-20T08:46:29.684553+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-20T08:45:54.654511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:45:54.654511+0000","last_clean_scrub_stamp":"2024-03-20T08:45:54.654511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T16:51:24.976022+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"64'2","reported_seq":38,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-20T08:46:29.606814+0000","last_change":"2024-03-20T08:45:53.655794+0000","last_active":"2024-03-20T08:46:29.606814+0000","last_peered":"2024-03-20T08:46:29.606814+0000","last_clean":"2024-03-20T08:46:29.606814+0000","last_became_active":"2024-03-20T08:45:49.628760+0000","last_became_peered":"2024-03-20T08:45:49.628760+0000","last_unstale":"2024-03-20T08:46:29.606814+0000","last_undegraded":"2024-03-20T08:46:29.606814+0000","last_fullsized":"2024-03-20T08:46:29.606814+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-20T08:45:48.603868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:45:48.603868+0000","last_clean_scrub_stamp":"2024-03-20T08:45:48.603868+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T15:41:27.543966+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0015906500000000001,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":29,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-20T08:46:29.658330+0000","last_change":"2024-03-20T08:45:49.631750+0000","last_active":"2024-03-20T08:46:29.658330+0000","last_peered":"2024-03-20T08:46:29.658330+0000","last_clean":"2024-03-20T08:46:29.658330+0000","last_became_active":"2024-03-20T08:45:49.630383+0000","last_became_peered":"2024-03-20T08:45:49.630383+0000","last_unstale":"2024-03-20T08:46:29.658330+0000","last_undegraded":"2024-03-20T08:46:29.658330+0000","last_fullsized":"2024-03-20T08:46:29.658330+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-20T08:45:48.603868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:45:48.603868+0000","last_clean_scrub_stamp":"2024-03-20T08:45:48.603868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T17:13:00.588163+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.4","version":"68'30","reported_seq":95,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-20T08:47:14.687608+0000","last_change":"2024-03-20T08:45:51.646326+0000","last_active":"2024-03-20T08:47:14.687608+0000","last_peered":"2024-03-20T08:47:14.687608+0000","last_clean":"2024-03-20T08:47:14.687608+0000","last_became_active":"2024-03-20T08:45:51.646030+0000","last_became_peered":"2024-03-20T08:45:51.646030+0000","last_unstale":"2024-03-20T08:47:14.687608+0000","last_undegraded":"2024-03-20T08:47:14.687608+0000","last_fullsized":"2024-03-20T08:47:14.687608+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-20T08:45:50.616760+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:45:50.616760+0000","last_clean_scrub_stamp":"2024-03-20T08:45:50.616760+0000","objects_scrubbed":0,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T15:45:40.271612+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":51,"num_read_kb":36,"num_write":26,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":21,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-20T08:46:29.611015+0000","last_change":"2024-03-20T08:45:53.655027+0000","last_active":"2024-03-20T08:46:29.611015+0000","last_peered":"2024-03-20T08:46:29.611015+0000","last_clean":"2024-03-20T08:46:29.611015+0000","last_became_active":"2024-03-20T08:45:53.654604+0000","last_became_peered":"2024-03-20T08:45:53.654604+0000","last_unstale":"2024-03-20T08:46:29.611015+0000","last_undegraded":"2024-03-20T08:46:29.611015+0000","last_fullsized":"2024-03-20T08:46:29.611015+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-20T08:45:52.630650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:45:52.630650+0000","last_clean_scrub_stamp":"2024-03-20T08:45:52.630650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T15:10:01.286030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":17,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-20T08:46:29.684821+0000","last_change":"2024-03-20T08:45:55.817347+0000","last_active":"2024-03-20T08:46:29.684821+0000","last_peered":"2024-03-20T08:46:29.684821+0000","last_clean":"2024-03-20T08:46:29.684821+0000","last_became_active":"2024-03-20T08:45:55.816689+0000","last_became_peered":"2024-03-20T08:45:55.816689+0000","last_unstale":"2024-03-20T08:46:29.684821+0000","last_undegraded":"2024-03-20T08:46:29.684821+0000","last_fullsized":"2024-03-20T08:46:29.684821+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-20T08:45:54.654511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:45:54.654511+0000","last_clean_scrub_stamp":"2024-03-20T08:45:54.654511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T16:03:23.078938+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":29,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-20T08:46:29.605660+0000","last_change":"2024-03-20T08:45:49.629091+0000","last_active":"2024-03-20T08:46:29.605660+0000","last_peered":"2024-03-20T08:46:29.605660+0000","last_clean":"2024-03-20T08:46:29.605660+0000","last_became_active":"2024-03-20T08:45:49.628769+0000","last_became_peered":"2024-03-20T08:45:49.628769+0000","last_unstale":"2024-03-20T08:46:29.605660+0000","last_undegraded":"2024-03-20T08:46:29.605660+0000","last_fullsized":"2024-03-20T08:46:29.605660+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-20T08:45:48.603868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:45:48.603868+0000","last_clean_scrub_stamp":"2024-03-20T08:45:48.603868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:30:36.931224+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":36,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-20T08:46:29.684117+0000","last_change":"2024-03-20T08:45:53.658005+0000","last_active":"2024-03-20T08:46:29.684117+0000","last_peered":"2024-03-20T08:46:29.684117+0000","last_clean":"2024-03-20T08:46:29.684117+0000","last_became_active":"2024-03-20T08:45:49.626638+0000","last_became_peered":"2024-03-20T08:45:49.626638+0000","last_unstale":"2024-03-20T08:46:29.684117+0000","last_undegraded":"2024-03-20T08:46:29.684117+0000","last_fullsized":"2024-03-20T08:46:29.684117+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-20T08:45:48.603868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:45:48.603868+0000","last_clean_scrub_stamp":"2024-03-20T08:45:48.603868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T09:43:03.860694+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0011759870000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"67'13","reported_seq":52,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-20T08:46:29.607988+0000","last_change":"2024-03-20T08:45:51.651666+0000","last_active":"2024-03-20T08:46:29.607988+0000","last_peered":"2024-03-20T08:46:29.607988+0000","last_clean":"2024-03-20T08:46:29.607988+0000","last_became_active":"2024-03-20T08:45:51.651322+0000","last_became_peered":"2024-03-20T08:45:51.651322+0000","last_unstale":"2024-03-20T08:46:29.607988+0000","last_undegraded":"2024-03-20T08:46:29.607988+0000","last_fullsized":"2024-03-20T08:46:29.607988+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-20T08:45:50.616760+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:45:50.616760+0000","last_clean_scrub_stamp":"2024-03-20T08:45:50.616760+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T19:58:27.420251+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":19,"num_write":16,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":21,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-20T08:46:29.684400+0000","last_change":"2024-03-20T08:45:53.666140+0000","last_active":"2024-03-20T08:46:29.684400+0000","last_peered":"2024-03-20T08:46:29.684400+0000","last_clean":"2024-03-20T08:46:29.684400+0000","last_became_active":"2024-03-20T08:45:53.665809+0000","last_became_peered":"2024-03-20T08:45:53.665809+0000","last_unstale":"2024-03-20T08:46:29.684400+0000","last_undegraded":"2024-03-20T08:46:29.684400+0000","last_fullsized":"2024-03-20T08:46:29.684400+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-20T08:45:52.630650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:45:52.630650+0000","last_clean_scrub_stamp":"2024-03-20T08:45:52.630650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T12:11:41.730622+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":17,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-20T08:46:29.607148+0000","last_change":"2024-03-20T08:45:55.691156+0000","last_active":"2024-03-20T08:46:29.607148+0000","last_peered":"2024-03-20T08:46:29.607148+0000","last_clean":"2024-03-20T08:46:29.607148+0000","last_became_active":"2024-03-20T08:45:55.690636+0000","last_became_peered":"2024-03-20T08:45:55.690636+0000","last_unstale":"2024-03-20T08:46:29.607148+0000","last_undegraded":"2024-03-20T08:46:29.607148+0000","last_fullsized":"2024-03-20T08:46:29.607148+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-20T08:45:54.654511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:45:54.654511+0000","last_clean_scrub_stamp":"2024-03-20T08:45:54.654511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T10:45:58.440667+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"69'188","reported_seq":337,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-20T08:46:31.703792+0000","last_change":"2024-03-20T08:45:35.514394+0000","last_active":"2024-03-20T08:46:31.703792+0000","last_peered":"2024-03-20T08:46:31.703792+0000","last_clean":"2024-03-20T08:46:31.703792+0000","last_became_active":"2024-03-20T08:45:35.513993+0000","last_became_peered":"2024-03-20T08:45:35.513993+0000","last_unstale":"2024-03-20T08:46:31.703792+0000","last_undegraded":"2024-03-20T08:46:31.703792+0000","last_fullsized":"2024-03-20T08:46:31.703792+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-20T08:43:22.115757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:43:22.115757+0000","last_clean_scrub_stamp":"2024-03-20T08:43:22.115757+0000","objects_scrubbed":0,"log_size":188,"log_dups_size":0,"ondisk_log_size":188,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T11:51:51.680611+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":242,"num_read_kb":333,"num_write":245,"num_write_kb":3982,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":29,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-20T08:46:29.686111+0000","last_change":"2024-03-20T08:45:49.633722+0000","last_active":"2024-03-20T08:46:29.686111+0000","last_peered":"2024-03-20T08:46:29.686111+0000","last_clean":"2024-03-20T08:46:29.686111+0000","last_became_active":"2024-03-20T08:45:49.633444+0000","last_became_peered":"2024-03-20T08:45:49.633444+0000","last_unstale":"2024-03-20T08:46:29.686111+0000","last_undegraded":"2024-03-20T08:46:29.686111+0000","last_fullsized":"2024-03-20T08:46:29.686111+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-20T08:45:48.603868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:45:48.603868+0000","last_clean_scrub_stamp":"2024-03-20T08:45:48.603868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T20:24:42.236163+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"67'16","reported_seq":66,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-20T08:47:14.687729+0000","last_change":"2024-03-20T08:45:51.645074+0000","last_active":"2024-03-20T08:47:14.687729+0000","last_peered":"2024-03-20T08:47:14.687729+0000","last_clean":"2024-03-20T08:47:14.687729+0000","last_became_active":"2024-03-20T08:45:51.644762+0000","last_became_peered":"2024-03-20T08:45:51.644762+0000","last_unstale":"2024-03-20T08:47:14.687729+0000","last_undegraded":"2024-03-20T08:47:14.687729+0000","last_fullsized":"2024-03-20T08:47:14.687729+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-20T08:45:50.616760+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:45:50.616760+0000","last_clean_scrub_stamp":"2024-03-20T08:45:50.616760+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T09:12:27.582168+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":25,"num_read_kb":15,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":21,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-20T08:46:29.683638+0000","last_change":"2024-03-20T08:45:53.666607+0000","last_active":"2024-03-20T08:46:29.683638+0000","last_peered":"2024-03-20T08:46:29.683638+0000","last_clean":"2024-03-20T08:46:29.683638+0000","last_became_active":"2024-03-20T08:45:53.666132+0000","last_became_peered":"2024-03-20T08:45:53.666132+0000","last_unstale":"2024-03-20T08:46:29.683638+0000","last_undegraded":"2024-03-20T08:46:29.683638+0000","last_fullsized":"2024-03-20T08:46:29.683638+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-20T08:45:52.630650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:45:52.630650+0000","last_clean_scrub_stamp":"2024-03-20T08:45:52.630650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T09:30:15.843001+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":17,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-20T08:46:29.606181+0000","last_change":"2024-03-20T08:45:55.691471+0000","last_active":"2024-03-20T08:46:29.606181+0000","last_peered":"2024-03-20T08:46:29.606181+0000","last_clean":"2024-03-20T08:46:29.606181+0000","last_became_active":"2024-03-20T08:45:55.690776+0000","last_became_peered":"2024-03-20T08:45:55.690776+0000","last_unstale":"2024-03-20T08:46:29.606181+0000","last_undegraded":"2024-03-20T08:46:29.606181+0000","last_fullsized":"2024-03-20T08:46:29.606181+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-20T08:45:54.654511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:45:54.654511+0000","last_clean_scrub_stamp":"2024-03-20T08:45:54.654511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:41:00.102045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":29,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-20T08:46:29.610362+0000","last_change":"2024-03-20T08:45:49.626166+0000","last_active":"2024-03-20T08:46:29.610362+0000","last_peered":"2024-03-20T08:46:29.610362+0000","last_clean":"2024-03-20T08:46:29.610362+0000","last_became_active":"2024-03-20T08:45:49.625892+0000","last_became_peered":"2024-03-20T08:45:49.625892+0000","last_unstale":"2024-03-20T08:46:29.610362+0000","last_undegraded":"2024-03-20T08:46:29.610362+0000","last_fullsized":"2024-03-20T08:46:29.610362+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-20T08:45:48.603868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:45:48.603868+0000","last_clean_scrub_stamp":"2024-03-20T08:45:48.603868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:14:03.125268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"68'5","reported_seq":124,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-20T08:47:19.017860+0000","last_change":"2024-03-20T08:45:53.658023+0000","last_active":"2024-03-20T08:47:19.017860+0000","last_peered":"2024-03-20T08:47:19.017860+0000","last_clean":"2024-03-20T08:47:19.017860+0000","last_became_active":"2024-03-20T08:45:49.634599+0000","last_became_peered":"2024-03-20T08:45:49.634599+0000","last_unstale":"2024-03-20T08:47:19.017860+0000","last_undegraded":"2024-03-20T08:47:19.017860+0000","last_fullsized":"2024-03-20T08:47:19.017860+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-20T08:45:48.603868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:45:48.603868+0000","last_clean_scrub_stamp":"2024-03-20T08:45:48.603868+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T12:27:18.966005+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0044392149999999998,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":85,"num_read_kb":80,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"67'12","reported_seq":43,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-20T08:46:29.610185+0000","last_change":"2024-03-20T08:45:51.649602+0000","last_active":"2024-03-20T08:46:29.610185+0000","last_peered":"2024-03-20T08:46:29.610185+0000","last_clean":"2024-03-20T08:46:29.610185+0000","last_became_active":"2024-03-20T08:45:51.649259+0000","last_became_peered":"2024-03-20T08:45:51.649259+0000","last_unstale":"2024-03-20T08:46:29.610185+0000","last_undegraded":"2024-03-20T08:46:29.610185+0000","last_fullsized":"2024-03-20T08:46:29.610185+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-20T08:45:50.616760+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:45:50.616760+0000","last_clean_scrub_stamp":"2024-03-20T08:45:50.616760+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T11:24:07.689854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":18,"num_read_kb":12,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":21,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-20T08:46:29.607394+0000","last_change":"2024-03-20T08:45:53.657977+0000","last_active":"2024-03-20T08:46:29.607394+0000","last_peered":"2024-03-20T08:46:29.607394+0000","last_clean":"2024-03-20T08:46:29.607394+0000","last_became_active":"2024-03-20T08:45:53.657640+0000","last_became_peered":"2024-03-20T08:45:53.657640+0000","last_unstale":"2024-03-20T08:46:29.607394+0000","last_undegraded":"2024-03-20T08:46:29.607394+0000","last_fullsized":"2024-03-20T08:46:29.607394+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-20T08:45:52.630650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:45:52.630650+0000","last_clean_scrub_stamp":"2024-03-20T08:45:52.630650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T08:46:00.972890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1","version":"67'1","reported_seq":18,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-20T08:46:29.605533+0000","last_change":"2024-03-20T08:45:55.815254+0000","last_active":"2024-03-20T08:46:29.605533+0000","last_peered":"2024-03-20T08:46:29.605533+0000","last_clean":"2024-03-20T08:46:29.605533+0000","last_became_active":"2024-03-20T08:45:55.814810+0000","last_became_peered":"2024-03-20T08:45:55.814810+0000","last_unstale":"2024-03-20T08:46:29.605533+0000","last_undegraded":"2024-03-20T08:46:29.605533+0000","last_fullsized":"2024-03-20T08:46:29.605533+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-20T08:45:54.654511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:45:54.654511+0000","last_clean_scrub_stamp":"2024-03-20T08:45:54.654511+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T18:39:56.308158+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":29,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-20T08:46:29.608270+0000","last_change":"2024-03-20T08:45:49.627368+0000","last_active":"2024-03-20T08:46:29.608270+0000","last_peered":"2024-03-20T08:46:29.608270+0000","last_clean":"2024-03-20T08:46:29.608270+0000","last_became_active":"2024-03-20T08:45:49.627035+0000","last_became_peered":"2024-03-20T08:45:49.627035+0000","last_unstale":"2024-03-20T08:46:29.608270+0000","last_undegraded":"2024-03-20T08:46:29.608270+0000","last_fullsized":"2024-03-20T08:46:29.608270+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-20T08:45:48.603868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:45:48.603868+0000","last_clean_scrub_stamp":"2024-03-20T08:45:48.603868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T19:58:15.422219+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.3","version":"67'19","reported_seq":61,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-20T08:46:29.611153+0000","last_change":"2024-03-20T08:45:51.642189+0000","last_active":"2024-03-20T08:46:29.611153+0000","last_peered":"2024-03-20T08:46:29.611153+0000","last_clean":"2024-03-20T08:46:29.611153+0000","last_became_active":"2024-03-20T08:45:51.641430+0000","last_became_peered":"2024-03-20T08:45:51.641430+0000","last_unstale":"2024-03-20T08:46:29.611153+0000","last_undegraded":"2024-03-20T08:46:29.611153+0000","last_fullsized":"2024-03-20T08:46:29.611153+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-20T08:45:50.616760+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:45:50.616760+0000","last_clean_scrub_stamp":"2024-03-20T08:45:50.616760+0000","objects_scrubbed":0,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T10:10:41.927770+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":39,"num_read_kb":25,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":21,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-20T08:46:29.684204+0000","last_change":"2024-03-20T08:45:53.660344+0000","last_active":"2024-03-20T08:46:29.684204+0000","last_peered":"2024-03-20T08:46:29.684204+0000","last_clean":"2024-03-20T08:46:29.684204+0000","last_became_active":"2024-03-20T08:45:53.660035+0000","last_became_peered":"2024-03-20T08:45:53.660035+0000","last_unstale":"2024-03-20T08:46:29.684204+0000","last_undegraded":"2024-03-20T08:46:29.684204+0000","last_fullsized":"2024-03-20T08:46:29.684204+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-20T08:45:52.630650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:45:52.630650+0000","last_clean_scrub_stamp":"2024-03-20T08:45:52.630650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:39:37.660656+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":17,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-20T08:46:29.609823+0000","last_change":"2024-03-20T08:45:55.693872+0000","last_active":"2024-03-20T08:46:29.609823+0000","last_peered":"2024-03-20T08:46:29.609823+0000","last_clean":"2024-03-20T08:46:29.609823+0000","last_became_active":"2024-03-20T08:45:55.693399+0000","last_became_peered":"2024-03-20T08:45:55.693399+0000","last_unstale":"2024-03-20T08:46:29.609823+0000","last_undegraded":"2024-03-20T08:46:29.609823+0000","last_fullsized":"2024-03-20T08:46:29.609823+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-20T08:45:54.654511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:45:54.654511+0000","last_clean_scrub_stamp":"2024-03-20T08:45:54.654511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T12:05:11.432672+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":29,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-20T08:46:29.607440+0000","last_change":"2024-03-20T08:45:49.631145+0000","last_active":"2024-03-20T08:46:29.607440+0000","last_peered":"2024-03-20T08:46:29.607440+0000","last_clean":"2024-03-20T08:46:29.607440+0000","last_became_active":"2024-03-20T08:45:49.630851+0000","last_became_peered":"2024-03-20T08:45:49.630851+0000","last_unstale":"2024-03-20T08:46:29.607440+0000","last_undegraded":"2024-03-20T08:46:29.607440+0000","last_fullsized":"2024-03-20T08:46:29.607440+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-20T08:45:48.603868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:45:48.603868+0000","last_clean_scrub_stamp":"2024-03-20T08:45:48.603868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T17:18:59.598326+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.2","version":"67'10","reported_seq":40,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-20T08:46:29.606339+0000","last_change":"2024-03-20T08:45:51.647715+0000","last_active":"2024-03-20T08:46:29.606339+0000","last_peered":"2024-03-20T08:46:29.606339+0000","last_clean":"2024-03-20T08:46:29.606339+0000","last_became_active":"2024-03-20T08:45:51.647382+0000","last_became_peered":"2024-03-20T08:45:51.647382+0000","last_unstale":"2024-03-20T08:46:29.606339+0000","last_undegraded":"2024-03-20T08:46:29.606339+0000","last_fullsized":"2024-03-20T08:46:29.606339+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-20T08:45:50.616760+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:45:50.616760+0000","last_clean_scrub_stamp":"2024-03-20T08:45:50.616760+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T12:14:30.581405+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3","version":"68'11","reported_seq":51,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-20T08:47:14.687567+0000","last_change":"2024-03-20T08:45:53.658746+0000","last_active":"2024-03-20T08:47:14.687567+0000","last_peered":"2024-03-20T08:47:14.687567+0000","last_clean":"2024-03-20T08:47:14.687567+0000","last_became_active":"2024-03-20T08:45:53.657489+0000","last_became_peered":"2024-03-20T08:45:53.657489+0000","last_unstale":"2024-03-20T08:47:14.687567+0000","last_undegraded":"2024-03-20T08:47:14.687567+0000","last_fullsized":"2024-03-20T08:47:14.687567+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-20T08:45:52.630650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:45:52.630650+0000","last_clean_scrub_stamp":"2024-03-20T08:45:52.630650+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T20:13:54.527052+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":17,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-20T08:46:29.684056+0000","last_change":"2024-03-20T08:45:55.817446+0000","last_active":"2024-03-20T08:46:29.684056+0000","last_peered":"2024-03-20T08:46:29.684056+0000","last_clean":"2024-03-20T08:46:29.684056+0000","last_became_active":"2024-03-20T08:45:55.817107+0000","last_became_peered":"2024-03-20T08:45:55.817107+0000","last_unstale":"2024-03-20T08:46:29.684056+0000","last_undegraded":"2024-03-20T08:46:29.684056+0000","last_fullsized":"2024-03-20T08:46:29.684056+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-20T08:45:54.654511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:45:54.654511+0000","last_clean_scrub_stamp":"2024-03-20T08:45:54.654511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T18:24:02.978084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.6","version":"60'1","reported_seq":30,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-20T08:46:29.610310+0000","last_change":"2024-03-20T08:45:49.623198+0000","last_active":"2024-03-20T08:46:29.610310+0000","last_peered":"2024-03-20T08:46:29.610310+0000","last_clean":"2024-03-20T08:46:29.610310+0000","last_became_active":"2024-03-20T08:45:49.622878+0000","last_became_peered":"2024-03-20T08:45:49.622878+0000","last_unstale":"2024-03-20T08:46:29.610310+0000","last_undegraded":"2024-03-20T08:46:29.610310+0000","last_fullsized":"2024-03-20T08:46:29.610310+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-20T08:45:48.603868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:45:48.603868+0000","last_clean_scrub_stamp":"2024-03-20T08:45:48.603868+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T12:47:16.777271+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1","version":"67'14","reported_seq":46,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-20T08:46:29.658705+0000","last_change":"2024-03-20T08:45:51.643180+0000","last_active":"2024-03-20T08:46:29.658705+0000","last_peered":"2024-03-20T08:46:29.658705+0000","last_clean":"2024-03-20T08:46:29.658705+0000","last_became_active":"2024-03-20T08:45:51.642854+0000","last_became_peered":"2024-03-20T08:45:51.642854+0000","last_unstale":"2024-03-20T08:46:29.658705+0000","last_undegraded":"2024-03-20T08:46:29.658705+0000","last_fullsized":"2024-03-20T08:46:29.658705+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-20T08:45:50.616760+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:45:50.616760+0000","last_clean_scrub_stamp":"2024-03-20T08:45:50.616760+0000","objects_scrubbed":0,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T12:52:53.657307+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":21,"num_read_kb":14,"num_write":14,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.0","version":"68'11","reported_seq":51,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-20T08:47:14.687561+0000","last_change":"2024-03-20T08:45:53.659549+0000","last_active":"2024-03-20T08:47:14.687561+0000","last_peered":"2024-03-20T08:47:14.687561+0000","last_clean":"2024-03-20T08:47:14.687561+0000","last_became_active":"2024-03-20T08:45:53.659203+0000","last_became_peered":"2024-03-20T08:45:53.659203+0000","last_unstale":"2024-03-20T08:47:14.687561+0000","last_undegraded":"2024-03-20T08:47:14.687561+0000","last_fullsized":"2024-03-20T08:47:14.687561+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-20T08:45:52.630650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:45:52.630650+0000","last_clean_scrub_stamp":"2024-03-20T08:45:52.630650+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T15:43:37.857631+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":17,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-20T08:46:29.658705+0000","last_change":"2024-03-20T08:45:55.690584+0000","last_active":"2024-03-20T08:46:29.658705+0000","last_peered":"2024-03-20T08:46:29.658705+0000","last_clean":"2024-03-20T08:46:29.658705+0000","last_became_active":"2024-03-20T08:45:55.690284+0000","last_became_peered":"2024-03-20T08:45:55.690284+0000","last_unstale":"2024-03-20T08:46:29.658705+0000","last_undegraded":"2024-03-20T08:46:29.658705+0000","last_fullsized":"2024-03-20T08:46:29.658705+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-20T08:45:54.654511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:45:54.654511+0000","last_clean_scrub_stamp":"2024-03-20T08:45:54.654511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T09:01:02.410099+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":29,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-20T08:46:29.685984+0000","last_change":"2024-03-20T08:45:49.633279+0000","last_active":"2024-03-20T08:46:29.685984+0000","last_peered":"2024-03-20T08:46:29.685984+0000","last_clean":"2024-03-20T08:46:29.685984+0000","last_became_active":"2024-03-20T08:45:49.633000+0000","last_became_peered":"2024-03-20T08:45:49.633000+0000","last_unstale":"2024-03-20T08:46:29.685984+0000","last_undegraded":"2024-03-20T08:46:29.685984+0000","last_fullsized":"2024-03-20T08:46:29.685984+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-20T08:45:48.603868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:45:48.603868+0000","last_clean_scrub_stamp":"2024-03-20T08:45:48.603868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T20:20:55.450167+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.0","version":"67'18","reported_seq":57,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-20T08:46:29.686025+0000","last_change":"2024-03-20T08:45:51.648290+0000","last_active":"2024-03-20T08:46:29.686025+0000","last_peered":"2024-03-20T08:46:29.686025+0000","last_clean":"2024-03-20T08:46:29.686025+0000","last_became_active":"2024-03-20T08:45:51.647996+0000","last_became_peered":"2024-03-20T08:45:51.647996+0000","last_unstale":"2024-03-20T08:46:29.686025+0000","last_undegraded":"2024-03-20T08:46:29.686025+0000","last_fullsized":"2024-03-20T08:46:29.686025+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-20T08:45:50.616760+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:45:50.616760+0000","last_clean_scrub_stamp":"2024-03-20T08:45:50.616760+0000","objects_scrubbed":0,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T11:09:28.409992+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":34,"num_read_kb":22,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":21,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-20T08:46:29.658565+0000","last_change":"2024-03-20T08:45:53.662892+0000","last_active":"2024-03-20T08:46:29.658565+0000","last_peered":"2024-03-20T08:46:29.658565+0000","last_clean":"2024-03-20T08:46:29.658565+0000","last_became_active":"2024-03-20T08:45:53.662308+0000","last_became_peered":"2024-03-20T08:45:53.662308+0000","last_unstale":"2024-03-20T08:46:29.658565+0000","last_undegraded":"2024-03-20T08:46:29.658565+0000","last_fullsized":"2024-03-20T08:46:29.658565+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-20T08:45:52.630650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:45:52.630650+0000","last_clean_scrub_stamp":"2024-03-20T08:45:52.630650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T15:50:00.452961+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":17,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-20T08:46:29.606822+0000","last_change":"2024-03-20T08:45:55.687291+0000","last_active":"2024-03-20T08:46:29.606822+0000","last_peered":"2024-03-20T08:46:29.606822+0000","last_clean":"2024-03-20T08:46:29.606822+0000","last_became_active":"2024-03-20T08:45:55.687016+0000","last_became_peered":"2024-03-20T08:45:55.687016+0000","last_unstale":"2024-03-20T08:46:29.606822+0000","last_undegraded":"2024-03-20T08:46:29.606822+0000","last_fullsized":"2024-03-20T08:46:29.606822+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-20T08:45:54.654511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:45:54.654511+0000","last_clean_scrub_stamp":"2024-03-20T08:45:54.654511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T20:15:41.744529+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.e","version":"68'11","reported_seq":51,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-20T08:47:14.687471+0000","last_change":"2024-03-20T08:45:53.655202+0000","last_active":"2024-03-20T08:47:14.687471+0000","last_peered":"2024-03-20T08:47:14.687471+0000","last_clean":"2024-03-20T08:47:14.687471+0000","last_became_active":"2024-03-20T08:45:53.654874+0000","last_became_peered":"2024-03-20T08:45:53.654874+0000","last_unstale":"2024-03-20T08:47:14.687471+0000","last_undegraded":"2024-03-20T08:47:14.687471+0000","last_fullsized":"2024-03-20T08:47:14.687471+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-20T08:45:52.630650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:45:52.630650+0000","last_clean_scrub_stamp":"2024-03-20T08:45:52.630650+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T11:08:48.289832+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":29,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-20T08:46:29.685614+0000","last_change":"2024-03-20T08:45:49.628572+0000","last_active":"2024-03-20T08:46:29.685614+0000","last_peered":"2024-03-20T08:46:29.685614+0000","last_clean":"2024-03-20T08:46:29.685614+0000","last_became_active":"2024-03-20T08:45:49.628236+0000","last_became_peered":"2024-03-20T08:45:49.628236+0000","last_unstale":"2024-03-20T08:46:29.685614+0000","last_undegraded":"2024-03-20T08:46:29.685614+0000","last_fullsized":"2024-03-20T08:46:29.685614+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-20T08:45:48.603868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:45:48.603868+0000","last_clean_scrub_stamp":"2024-03-20T08:45:48.603868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:42:36.383174+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"67'15","reported_seq":50,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-20T08:46:29.607854+0000","last_change":"2024-03-20T08:45:51.649560+0000","last_active":"2024-03-20T08:46:29.607854+0000","last_peered":"2024-03-20T08:46:29.607854+0000","last_clean":"2024-03-20T08:46:29.607854+0000","last_became_active":"2024-03-20T08:45:51.649096+0000","last_became_peered":"2024-03-20T08:45:51.649096+0000","last_unstale":"2024-03-20T08:46:29.607854+0000","last_undegraded":"2024-03-20T08:46:29.607854+0000","last_fullsized":"2024-03-20T08:46:29.607854+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-20T08:45:50.616760+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:45:50.616760+0000","last_clean_scrub_stamp":"2024-03-20T08:45:50.616760+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T10:10:45.711033+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":17,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-20T08:46:29.686902+0000","last_change":"2024-03-20T08:45:55.816439+0000","last_active":"2024-03-20T08:46:29.686902+0000","last_peered":"2024-03-20T08:46:29.686902+0000","last_clean":"2024-03-20T08:46:29.686902+0000","last_became_active":"2024-03-20T08:45:55.814953+0000","last_became_peered":"2024-03-20T08:45:55.814953+0000","last_unstale":"2024-03-20T08:46:29.686902+0000","last_undegraded":"2024-03-20T08:46:29.686902+0000","last_fullsized":"2024-03-20T08:46:29.686902+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-20T08:45:54.654511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:45:54.654511+0000","last_clean_scrub_stamp":"2024-03-20T08:45:54.654511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T12:34:38.123915+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":21,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-20T08:46:29.607337+0000","last_change":"2024-03-20T08:45:53.665023+0000","last_active":"2024-03-20T08:46:29.607337+0000","last_peered":"2024-03-20T08:46:29.607337+0000","last_clean":"2024-03-20T08:46:29.607337+0000","last_became_active":"2024-03-20T08:45:53.664715+0000","last_became_peered":"2024-03-20T08:45:53.664715+0000","last_unstale":"2024-03-20T08:46:29.607337+0000","last_undegraded":"2024-03-20T08:46:29.607337+0000","last_fullsized":"2024-03-20T08:46:29.607337+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-20T08:45:52.630650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:45:52.630650+0000","last_clean_scrub_stamp":"2024-03-20T08:45:52.630650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T12:28:35.702350+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":29,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-20T08:46:29.658713+0000","last_change":"2024-03-20T08:45:49.631944+0000","last_active":"2024-03-20T08:46:29.658713+0000","last_peered":"2024-03-20T08:46:29.658713+0000","last_clean":"2024-03-20T08:46:29.658713+0000","last_became_active":"2024-03-20T08:45:49.629022+0000","last_became_peered":"2024-03-20T08:45:49.629022+0000","last_unstale":"2024-03-20T08:46:29.658713+0000","last_undegraded":"2024-03-20T08:46:29.658713+0000","last_fullsized":"2024-03-20T08:46:29.658713+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-20T08:45:48.603868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:45:48.603868+0000","last_clean_scrub_stamp":"2024-03-20T08:45:48.603868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:28:00.389627+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"67'11","reported_seq":44,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-20T08:46:29.659311+0000","last_change":"2024-03-20T08:45:51.648565+0000","last_active":"2024-03-20T08:46:29.659311+0000","last_peered":"2024-03-20T08:46:29.659311+0000","last_clean":"2024-03-20T08:46:29.659311+0000","last_became_active":"2024-03-20T08:45:51.648255+0000","last_became_peered":"2024-03-20T08:45:51.648255+0000","last_unstale":"2024-03-20T08:46:29.659311+0000","last_undegraded":"2024-03-20T08:46:29.659311+0000","last_fullsized":"2024-03-20T08:46:29.659311+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-20T08:45:50.616760+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:45:50.616760+0000","last_clean_scrub_stamp":"2024-03-20T08:45:50.616760+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T15:47:01.342506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":17,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-20T08:46:29.684706+0000","last_change":"2024-03-20T08:45:55.690766+0000","last_active":"2024-03-20T08:46:29.684706+0000","last_peered":"2024-03-20T08:46:29.684706+0000","last_clean":"2024-03-20T08:46:29.684706+0000","last_became_active":"2024-03-20T08:45:55.690354+0000","last_became_peered":"2024-03-20T08:45:55.690354+0000","last_unstale":"2024-03-20T08:46:29.684706+0000","last_undegraded":"2024-03-20T08:46:29.684706+0000","last_fullsized":"2024-03-20T08:46:29.684706+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-20T08:45:54.654511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:45:54.654511+0000","last_clean_scrub_stamp":"2024-03-20T08:45:54.654511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T08:58:43.782547+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":21,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-20T08:46:29.608549+0000","last_change":"2024-03-20T08:45:53.662905+0000","last_active":"2024-03-20T08:46:29.608549+0000","last_peered":"2024-03-20T08:46:29.608549+0000","last_clean":"2024-03-20T08:46:29.608549+0000","last_became_active":"2024-03-20T08:45:53.662390+0000","last_became_peered":"2024-03-20T08:45:53.662390+0000","last_unstale":"2024-03-20T08:46:29.608549+0000","last_undegraded":"2024-03-20T08:46:29.608549+0000","last_fullsized":"2024-03-20T08:46:29.608549+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-20T08:45:52.630650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:45:52.630650+0000","last_clean_scrub_stamp":"2024-03-20T08:45:52.630650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:08:02.985347+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":29,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-20T08:46:29.683344+0000","last_change":"2024-03-20T08:45:49.627755+0000","last_active":"2024-03-20T08:46:29.683344+0000","last_peered":"2024-03-20T08:46:29.683344+0000","last_clean":"2024-03-20T08:46:29.683344+0000","last_became_active":"2024-03-20T08:45:49.627529+0000","last_became_peered":"2024-03-20T08:45:49.627529+0000","last_unstale":"2024-03-20T08:46:29.683344+0000","last_undegraded":"2024-03-20T08:46:29.683344+0000","last_fullsized":"2024-03-20T08:46:29.683344+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-20T08:45:48.603868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:45:48.603868+0000","last_clean_scrub_stamp":"2024-03-20T08:45:48.603868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T16:56:35.899277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.d","version":"67'17","reported_seq":53,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-20T08:46:29.659327+0000","last_change":"2024-03-20T08:45:51.653119+0000","last_active":"2024-03-20T08:46:29.659327+0000","last_peered":"2024-03-20T08:46:29.659327+0000","last_clean":"2024-03-20T08:46:29.659327+0000","last_became_active":"2024-03-20T08:45:51.652808+0000","last_became_peered":"2024-03-20T08:45:51.652808+0000","last_unstale":"2024-03-20T08:46:29.659327+0000","last_undegraded":"2024-03-20T08:46:29.659327+0000","last_fullsized":"2024-03-20T08:46:29.659327+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-20T08:45:50.616760+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:45:50.616760+0000","last_clean_scrub_stamp":"2024-03-20T08:45:50.616760+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T10:13:15.965338+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":19,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":17,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-20T08:46:29.659501+0000","last_change":"2024-03-20T08:45:55.687136+0000","last_active":"2024-03-20T08:46:29.659501+0000","last_peered":"2024-03-20T08:46:29.659501+0000","last_clean":"2024-03-20T08:46:29.659501+0000","last_became_active":"2024-03-20T08:45:55.686739+0000","last_became_peered":"2024-03-20T08:45:55.686739+0000","last_unstale":"2024-03-20T08:46:29.659501+0000","last_undegraded":"2024-03-20T08:46:29.659501+0000","last_fullsized":"2024-03-20T08:46:29.659501+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-20T08:45:54.654511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:45:54.654511+0000","last_clean_scrub_stamp":"2024-03-20T08:45:54.654511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T20:00:48.398479+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"68'11","reported_seq":54,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-20T08:47:14.687568+0000","last_change":"2024-03-20T08:45:53.666994+0000","last_active":"2024-03-20T08:47:14.687568+0000","last_peered":"2024-03-20T08:47:14.687568+0000","last_clean":"2024-03-20T08:47:14.687568+0000","last_became_active":"2024-03-20T08:45:53.666656+0000","last_became_peered":"2024-03-20T08:45:53.666656+0000","last_unstale":"2024-03-20T08:47:14.687568+0000","last_undegraded":"2024-03-20T08:47:14.687568+0000","last_fullsized":"2024-03-20T08:47:14.687568+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-20T08:45:52.630650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:45:52.630650+0000","last_clean_scrub_stamp":"2024-03-20T08:45:52.630650+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:37:42.431488+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":29,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-20T08:46:29.686564+0000","last_change":"2024-03-20T08:45:49.630406+0000","last_active":"2024-03-20T08:46:29.686564+0000","last_peered":"2024-03-20T08:46:29.686564+0000","last_clean":"2024-03-20T08:46:29.686564+0000","last_became_active":"2024-03-20T08:45:49.630108+0000","last_became_peered":"2024-03-20T08:45:49.630108+0000","last_unstale":"2024-03-20T08:46:29.686564+0000","last_undegraded":"2024-03-20T08:46:29.686564+0000","last_fullsized":"2024-03-20T08:46:29.686564+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-20T08:45:48.603868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:45:48.603868+0000","last_clean_scrub_stamp":"2024-03-20T08:45:48.603868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T13:17:21.499429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"67'10","reported_seq":40,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-20T08:46:29.659096+0000","last_change":"2024-03-20T08:45:51.646151+0000","last_active":"2024-03-20T08:46:29.659096+0000","last_peered":"2024-03-20T08:46:29.659096+0000","last_clean":"2024-03-20T08:46:29.659096+0000","last_became_active":"2024-03-20T08:45:51.645785+0000","last_became_peered":"2024-03-20T08:45:51.645785+0000","last_unstale":"2024-03-20T08:46:29.659096+0000","last_undegraded":"2024-03-20T08:46:29.659096+0000","last_fullsized":"2024-03-20T08:46:29.659096+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-20T08:45:50.616760+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:45:50.616760+0000","last_clean_scrub_stamp":"2024-03-20T08:45:50.616760+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T20:03:40.452309+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":383,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1149,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":88,"ondisk_log_size":88,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":701,"num_read_kb":458,"num_write":417,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":395,"ondisk_log_size":395,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1531,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":34,"num_read_kb":34,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4593,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":96,"acting":96,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":242,"num_read_kb":333,"num_write":245,"num_write_kb":3982,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":188,"ondisk_log_size":188,"up":3,"acting":3,"num_store_stats":3},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":85,"num_read_kb":80,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":9,"acting":9,"num_store_stats":7}],"osd_stats":[{"osd":7,"up_from":54,"seq":231928234008,"num_pgs":53,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93741056,"kb_used":32516,"kb_used_data":1340,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93708540,"statfs":{"total":95990841344,"available":95957544960,"internally_reserved":0,"allocated":1372160,"data_stored":909424,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":46,"seq":197568495645,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93741056,"kb_used":32492,"kb_used_data":1316,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93708564,"statfs":{"total":95990841344,"available":95957569536,"internally_reserved":0,"allocated":1347584,"data_stored":907038,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574903,"num_pgs":57,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93741056,"kb_used":31984,"kb_used_data":804,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93709072,"statfs":{"total":95990841344,"available":95958089728,"internally_reserved":0,"allocated":823296,"data_stored":319366,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738429,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93741056,"kb_used":28352,"kb_used_data":1336,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93712704,"statfs":{"total":95990841344,"available":95961808896,"internally_reserved":0,"allocated":1368064,"data_stored":907819,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411379,"num_pgs":36,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93741056,"kb_used":27752,"kb_used_data":732,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93713304,"statfs":{"total":95990841344,"available":95962423296,"internally_reserved":0,"allocated":749568,"data_stored":316785,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182445,"num_pgs":56,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93741056,"kb_used":31952,"kb_used_data":772,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93709104,"statfs":{"total":95990841344,"available":95958122496,"internally_reserved":0,"allocated":790528,"data_stored":317984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":30,"seq":128849018919,"num_pgs":58,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93741056,"kb_used":27784,"kb_used_data":764,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93713272,"statfs":{"total":95990841344,"available":95962390528,"internally_reserved":0,"allocated":782336,"data_stored":317698,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757282,"num_pgs":47,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93741056,"kb_used":31920,"kb_used_data":740,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93709136,"statfs":{"total":95990841344,"available":95958155264,"internally_reserved":0,"allocated":757760,"data_stored":317894,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":138,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1003,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":482,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":620,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":911,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1439,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":90112,"data_stored":2338,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":798,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":990,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-20T08:47:22.800 INFO:tasks.cephadm.ceph_manager.ceph:clean! 2024-03-20T08:47:22.801 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-03-20T08:47:22.801 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy 2024-03-20T08:47:22.801 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 4c3f3eba-e695-11ee-95c9-87774f69a715 -- ceph health --format=json 2024-03-20T08:47:22.889 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:47:22 smithi194 ceph-mon[27237]: pgmap v29: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-20T08:47:22.889 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:47:22 smithi194 ceph-mon[27237]: from='client.14787 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-20T08:47:23.046 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/4c3f3eba-e695-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T08:47:23.172 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:47:22 smithi183 ceph-mon[31860]: pgmap v29: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-20T08:47:23.172 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:47:22 smithi183 ceph-mon[31860]: from='client.14787 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-20T08:47:23.172 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:47:22 smithi183 ceph-mon[25285]: pgmap v29: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-20T08:47:23.173 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:47:22 smithi183 ceph-mon[25285]: from='client.14787 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-20T08:47:23.474 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Mar 20 08:47:23 smithi194 podman[51812]: 2024-03-20 08:47:23.376553428 +0000 UTC m=+0.585728417 container start 98413bced1ff941af42ed1cd3fa177f8ed0baf311366b04e93983b1087f7462d (image=quay.io/ceph/ceph-grafana:9.4.12, name=ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-grafana-a, distribution-scope=public, io.k8s.display-name=Red Hat Universal Base Image 8, build-date=2022-04-29T11:48:11.055897, name=ubi8, version=8.5, vcs-type=git, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, vendor=Red Hat, Inc., com.redhat.component=ubi8-container, release=239.1651231664, io.openshift.tags=base rhel8, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-239.1651231664, architecture=x86_64, maintainer=Paul Cuzner , com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, io.openshift.expose-services=, summary=Grafana Container configured for Ceph mgr/dashboard integration, io.buildah.version=1.32.0, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., com.redhat.build-host=cpt-1007.osbs.prod.upshift.rdu2.redhat.com, description=Ceph Grafana Container) 2024-03-20T08:47:23.474 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Mar 20 08:47:23 smithi194 bash[51812]: 98413bced1ff941af42ed1cd3fa177f8ed0baf311366b04e93983b1087f7462d 2024-03-20T08:47:23.725 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Mar 20 08:47:23 smithi194 systemd[1]: Started Ceph grafana.a for 4c3f3eba-e695-11ee-95c9-87774f69a715. 2024-03-20T08:47:23.725 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Mar 20 08:47:23 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-grafana-a[51839]: logger=settings t=2024-03-20T08:47:23.527330643Z level=info msg="Starting Grafana" version=9.4.12 commit=37ee25ba46 branch=HEAD compiled=2023-05-22T15:58:05Z 2024-03-20T08:47:23.725 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Mar 20 08:47:23 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-grafana-a[51839]: logger=settings t=2024-03-20T08:47:23.527516188Z level=warn msg="\"sentry\" frontend logging provider is deprecated and will be removed in the next major version. Use \"grafana\" provider instead." 2024-03-20T08:47:23.725 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Mar 20 08:47:23 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-grafana-a[51839]: logger=settings t=2024-03-20T08:47:23.527534802Z level=info msg="Config loaded from" file=/usr/share/grafana/conf/defaults.ini 2024-03-20T08:47:23.725 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Mar 20 08:47:23 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-grafana-a[51839]: logger=settings t=2024-03-20T08:47:23.527543347Z level=info msg="Config loaded from" file=/etc/grafana/grafana.ini 2024-03-20T08:47:23.726 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Mar 20 08:47:23 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-grafana-a[51839]: logger=settings t=2024-03-20T08:47:23.52754994Z level=info msg="Config overridden from Environment variable" var="GF_PATHS_DATA=/var/lib/grafana" 2024-03-20T08:47:23.726 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Mar 20 08:47:23 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-grafana-a[51839]: logger=settings t=2024-03-20T08:47:23.527556803Z level=info msg="Config overridden from Environment variable" var="GF_PATHS_LOGS=/var/log/grafana" 2024-03-20T08:47:23.726 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Mar 20 08:47:23 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-grafana-a[51839]: logger=settings t=2024-03-20T08:47:23.527562573Z level=info msg="Config overridden from Environment variable" var="GF_PATHS_PLUGINS=/var/lib/grafana/plugins" 2024-03-20T08:47:23.726 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Mar 20 08:47:23 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-grafana-a[51839]: logger=settings t=2024-03-20T08:47:23.527568308Z level=info msg="Config overridden from Environment variable" var="GF_PATHS_PROVISIONING=/etc/grafana/provisioning" 2024-03-20T08:47:23.726 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Mar 20 08:47:23 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-grafana-a[51839]: logger=settings t=2024-03-20T08:47:23.527574205Z level=info msg="Path Home" path=/usr/share/grafana 2024-03-20T08:47:23.726 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Mar 20 08:47:23 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-grafana-a[51839]: logger=settings t=2024-03-20T08:47:23.527579881Z level=info msg="Path Data" path=/var/lib/grafana 2024-03-20T08:47:23.726 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Mar 20 08:47:23 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-grafana-a[51839]: logger=settings t=2024-03-20T08:47:23.527585346Z level=info msg="Path Logs" path=/var/log/grafana 2024-03-20T08:47:23.726 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Mar 20 08:47:23 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-grafana-a[51839]: logger=settings t=2024-03-20T08:47:23.527590665Z level=info msg="Path Plugins" path=/var/lib/grafana/plugins 2024-03-20T08:47:23.726 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Mar 20 08:47:23 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-grafana-a[51839]: logger=settings t=2024-03-20T08:47:23.527596276Z level=info msg="Path Provisioning" path=/etc/grafana/provisioning 2024-03-20T08:47:23.727 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Mar 20 08:47:23 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-grafana-a[51839]: logger=settings t=2024-03-20T08:47:23.527601845Z level=info msg="App mode production" 2024-03-20T08:47:23.727 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Mar 20 08:47:23 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-grafana-a[51839]: logger=sqlstore t=2024-03-20T08:47:23.52764474Z level=info msg="Connecting to DB" dbtype=sqlite3 2024-03-20T08:47:23.727 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Mar 20 08:47:23 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-grafana-a[51839]: logger=sqlstore t=2024-03-20T08:47:23.527660157Z level=warn msg="SQLite database file has broader permissions than it should" path=/var/lib/grafana/grafana.db mode=-rw-r--r-- expected=-rw-r----- 2024-03-20T08:47:23.727 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Mar 20 08:47:23 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-grafana-a[51839]: logger=migrator t=2024-03-20T08:47:23.528521017Z level=info msg="Starting DB migrations" 2024-03-20T08:47:23.727 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Mar 20 08:47:23 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-grafana-a[51839]: logger=migrator t=2024-03-20T08:47:23.52859509Z level=info msg="Executing migration" id="create migration_log table" 2024-03-20T08:47:23.727 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Mar 20 08:47:23 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-grafana-a[51839]: logger=migrator t=2024-03-20T08:47:23.529132005Z level=info msg="Executing migration" id="create user table" 2024-03-20T08:47:23.727 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Mar 20 08:47:23 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-grafana-a[51839]: logger=migrator t=2024-03-20T08:47:23.529629958Z level=info msg="Executing migration" id="add unique index user.login" 2024-03-20T08:47:23.727 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Mar 20 08:47:23 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-grafana-a[51839]: logger=migrator t=2024-03-20T08:47:23.530129046Z level=info msg="Executing migration" id="add unique index user.email" 2024-03-20T08:47:23.727 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Mar 20 08:47:23 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-grafana-a[51839]: logger=migrator t=2024-03-20T08:47:23.530570095Z level=info msg="Executing migration" id="drop index UQE_user_login - v1" 2024-03-20T08:47:23.728 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Mar 20 08:47:23 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-grafana-a[51839]: logger=migrator t=2024-03-20T08:47:23.531318202Z level=info msg="Executing migration" id="drop index UQE_user_email - v1" 2024-03-20T08:47:23.728 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Mar 20 08:47:23 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-grafana-a[51839]: logger=migrator t=2024-03-20T08:47:23.531814592Z level=info msg="Executing migration" id="Rename table user to user_v1 - v1" 2024-03-20T08:47:23.728 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Mar 20 08:47:23 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-grafana-a[51839]: logger=migrator t=2024-03-20T08:47:23.532910278Z level=info msg="Executing migration" id="create user table v2" 2024-03-20T08:47:23.728 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Mar 20 08:47:23 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-grafana-a[51839]: logger=migrator t=2024-03-20T08:47:23.533330562Z level=info msg="Executing migration" id="create index UQE_user_login - v2" 2024-03-20T08:47:23.728 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Mar 20 08:47:23 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-grafana-a[51839]: logger=migrator t=2024-03-20T08:47:23.533715115Z level=info msg="Executing migration" id="create index UQE_user_email - v2" 2024-03-20T08:47:23.728 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Mar 20 08:47:23 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-grafana-a[51839]: logger=migrator t=2024-03-20T08:47:23.534128474Z level=info msg="Executing migration" id="copy data_source v1 to v2" 2024-03-20T08:47:23.728 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Mar 20 08:47:23 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-grafana-a[51839]: logger=migrator t=2024-03-20T08:47:23.53447376Z level=info msg="Executing migration" id="Drop old table user_v1" 2024-03-20T08:47:23.728 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Mar 20 08:47:23 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-grafana-a[51839]: logger=migrator t=2024-03-20T08:47:23.534786152Z level=info msg="Executing migration" id="Add column help_flags1 to user table" 2024-03-20T08:47:23.729 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Mar 20 08:47:23 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-grafana-a[51839]: logger=migrator t=2024-03-20T08:47:23.535203372Z level=info msg="Executing migration" id="Update user table charset" 2024-03-20T08:47:23.729 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Mar 20 08:47:23 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-grafana-a[51839]: logger=migrator t=2024-03-20T08:47:23.535495157Z level=info msg="Executing migration" id="Add last_seen_at column to user" 2024-03-20T08:47:23.729 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Mar 20 08:47:23 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-grafana-a[51839]: logger=migrator t=2024-03-20T08:47:23.535879991Z level=info msg="Executing migration" id="Add missing user data" 2024-03-20T08:47:23.729 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Mar 20 08:47:23 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-grafana-a[51839]: logger=migrator t=2024-03-20T08:47:23.536249376Z level=info msg="Executing migration" id="Add is_disabled column to user" 2024-03-20T08:47:23.729 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Mar 20 08:47:23 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-grafana-a[51839]: logger=migrator t=2024-03-20T08:47:23.53664557Z level=info msg="Executing migration" id="Add index user.login/user.email" 2024-03-20T08:47:23.729 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Mar 20 08:47:23 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-grafana-a[51839]: logger=migrator t=2024-03-20T08:47:23.537065825Z level=info msg="Executing migration" id="Add is_service_account column to user" 2024-03-20T08:47:23.729 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Mar 20 08:47:23 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-grafana-a[51839]: logger=migrator t=2024-03-20T08:47:23.537484327Z level=info msg="Executing migration" id="Update is_service_account column to nullable" 2024-03-20T08:47:23.729 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Mar 20 08:47:23 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-grafana-a[51839]: logger=migrator t=2024-03-20T08:47:23.539005796Z level=info msg="Executing migration" id="create temp user table v1-7" 2024-03-20T08:47:23.731 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Mar 20 08:47:23 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-grafana-a[51839]: logger=migrator t=2024-03-20T08:47:23.539531311Z level=info msg="Executing migration" id="create index IDX_temp_user_email - v1-7" 2024-03-20T08:47:23.731 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Mar 20 08:47:23 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-grafana-a[51839]: logger=migrator t=2024-03-20T08:47:23.540066229Z level=info msg="Executing migration" id="create index IDX_temp_user_org_id - v1-7" 2024-03-20T08:47:23.732 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Mar 20 08:47:23 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-grafana-a[51839]: logger=migrator t=2024-03-20T08:47:23.540559348Z level=info msg="Executing migration" id="create index IDX_temp_user_code - v1-7" 2024-03-20T08:47:23.732 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Mar 20 08:47:23 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-grafana-a[51839]: logger=migrator t=2024-03-20T08:47:23.540980822Z level=info msg="Executing migration" id="create index IDX_temp_user_status - v1-7" 2024-03-20T08:47:23.732 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Mar 20 08:47:23 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-grafana-a[51839]: logger=migrator t=2024-03-20T08:47:23.541844349Z level=info msg="Executing migration" id="Update temp_user table charset" 2024-03-20T08:47:23.732 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Mar 20 08:47:23 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-grafana-a[51839]: logger=migrator t=2024-03-20T08:47:23.542679973Z level=info msg="Executing migration" id="drop index IDX_temp_user_email - v1" 2024-03-20T08:47:23.732 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Mar 20 08:47:23 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-grafana-a[51839]: logger=migrator t=2024-03-20T08:47:23.543530819Z level=info msg="Executing migration" id="drop index IDX_temp_user_org_id - v1" 2024-03-20T08:47:23.732 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Mar 20 08:47:23 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-grafana-a[51839]: logger=migrator t=2024-03-20T08:47:23.544384361Z level=info msg="Executing migration" id="drop index IDX_temp_user_code - v1" 2024-03-20T08:47:23.732 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Mar 20 08:47:23 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-grafana-a[51839]: logger=migrator t=2024-03-20T08:47:23.54527506Z level=info msg="Executing migration" id="drop index IDX_temp_user_status - v1" 2024-03-20T08:47:23.732 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Mar 20 08:47:23 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-grafana-a[51839]: logger=migrator t=2024-03-20T08:47:23.546223782Z level=info msg="Executing migration" id="Rename table temp_user to temp_user_tmp_qwerty - v1" 2024-03-20T08:47:23.732 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Mar 20 08:47:23 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-grafana-a[51839]: logger=migrator t=2024-03-20T08:47:23.547067908Z level=info msg="Executing migration" id="create temp_user v2" 2024-03-20T08:47:23.733 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Mar 20 08:47:23 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-grafana-a[51839]: logger=migrator t=2024-03-20T08:47:23.550778954Z level=info msg="Executing migration" id="create index IDX_temp_user_email - v2" 2024-03-20T08:47:23.733 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Mar 20 08:47:23 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-grafana-a[51839]: logger=migrator t=2024-03-20T08:47:23.552787431Z level=info msg="Executing migration" id="create index IDX_temp_user_org_id - v2" 2024-03-20T08:47:23.733 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Mar 20 08:47:23 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-grafana-a[51839]: logger=migrator t=2024-03-20T08:47:23.555761845Z level=info msg="Executing migration" id="create index IDX_temp_user_code - v2" 2024-03-20T08:47:23.733 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Mar 20 08:47:23 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-grafana-a[51839]: logger=migrator t=2024-03-20T08:47:23.558575624Z level=info msg="Executing migration" id="create index IDX_temp_user_status - v2" 2024-03-20T08:47:23.733 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Mar 20 08:47:23 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-grafana-a[51839]: logger=migrator t=2024-03-20T08:47:23.561259603Z level=info msg="Executing migration" id="copy temp_user v1 to v2" 2024-03-20T08:47:23.733 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Mar 20 08:47:23 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-grafana-a[51839]: logger=migrator t=2024-03-20T08:47:23.564104997Z level=info msg="Executing migration" id="drop temp_user_tmp_qwerty" 2024-03-20T08:47:23.733 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Mar 20 08:47:23 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-grafana-a[51839]: logger=migrator t=2024-03-20T08:47:23.564454862Z level=info msg="Executing migration" id="Set created for temp users that will otherwise prematurely expire" 2024-03-20T08:47:23.733 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Mar 20 08:47:23 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-grafana-a[51839]: logger=migrator t=2024-03-20T08:47:23.567144998Z level=info msg="Executing migration" id="create star table" 2024-03-20T08:47:23.734 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Mar 20 08:47:23 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-grafana-a[51839]: logger=migrator t=2024-03-20T08:47:23.569849117Z level=info msg="Executing migration" id="add unique index star.user_id_dashboard_id" 2024-03-20T08:47:23.734 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Mar 20 08:47:23 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-grafana-a[51839]: logger=migrator t=2024-03-20T08:47:23.575548186Z level=info msg="Executing migration" id="create org table v1" 2024-03-20T08:47:23.734 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Mar 20 08:47:23 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-grafana-a[51839]: logger=migrator t=2024-03-20T08:47:23.576002949Z level=info msg="Executing migration" id="create index UQE_org_name - v1" 2024-03-20T08:47:23.734 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Mar 20 08:47:23 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-grafana-a[51839]: logger=migrator t=2024-03-20T08:47:23.576982084Z level=info msg="Executing migration" id="create org_user table v1" 2024-03-20T08:47:23.734 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Mar 20 08:47:23 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-grafana-a[51839]: logger=migrator t=2024-03-20T08:47:23.577813391Z level=info msg="Executing migration" id="create index IDX_org_user_org_id - v1" 2024-03-20T08:47:23.734 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Mar 20 08:47:23 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-grafana-a[51839]: logger=migrator t=2024-03-20T08:47:23.578398735Z level=info msg="Executing migration" id="create index UQE_org_user_org_id_user_id - v1" 2024-03-20T08:47:23.734 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Mar 20 08:47:23 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-grafana-a[51839]: logger=migrator t=2024-03-20T08:47:23.579124331Z level=info msg="Executing migration" id="create index IDX_org_user_user_id - v1" 2024-03-20T08:47:23.734 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Mar 20 08:47:23 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-grafana-a[51839]: logger=migrator t=2024-03-20T08:47:23.57997671Z level=info msg="Executing migration" id="Update org table charset" 2024-03-20T08:47:23.734 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Mar 20 08:47:23 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-grafana-a[51839]: logger=migrator t=2024-03-20T08:47:23.580596223Z level=info msg="Executing migration" id="Update org_user table charset" 2024-03-20T08:47:23.734 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Mar 20 08:47:23 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-grafana-a[51839]: logger=migrator t=2024-03-20T08:47:23.581322771Z level=info msg="Executing migration" id="Migrate all Read Only Viewers to Viewers" 2024-03-20T08:47:23.734 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Mar 20 08:47:23 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-grafana-a[51839]: logger=migrator t=2024-03-20T08:47:23.581681785Z level=info msg="Executing migration" id="create dashboard table" 2024-03-20T08:47:23.735 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Mar 20 08:47:23 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-grafana-a[51839]: logger=migrator t=2024-03-20T08:47:23.582081813Z level=info msg="Executing migration" id="add index dashboard.account_id" 2024-03-20T08:47:23.735 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Mar 20 08:47:23 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-grafana-a[51839]: logger=migrator t=2024-03-20T08:47:23.582825003Z level=info msg="Executing migration" id="add unique index dashboard_account_id_slug" 2024-03-20T08:47:23.735 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Mar 20 08:47:23 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-grafana-a[51839]: logger=migrator t=2024-03-20T08:47:23.583584214Z level=info msg="Executing migration" id="create dashboard_tag table" 2024-03-20T08:47:23.735 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Mar 20 08:47:23 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-grafana-a[51839]: logger=migrator t=2024-03-20T08:47:23.584327247Z level=info msg="Executing migration" id="add unique index dashboard_tag.dasboard_id_term" 2024-03-20T08:47:23.735 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Mar 20 08:47:23 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-grafana-a[51839]: logger=migrator t=2024-03-20T08:47:23.585087948Z level=info msg="Executing migration" id="drop index UQE_dashboard_tag_dashboard_id_term - v1" 2024-03-20T08:47:23.735 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Mar 20 08:47:23 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-grafana-a[51839]: logger=migrator t=2024-03-20T08:47:23.585871629Z level=info msg="Executing migration" id="Rename table dashboard to dashboard_v1 - v1" 2024-03-20T08:47:23.735 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Mar 20 08:47:23 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-grafana-a[51839]: logger=migrator t=2024-03-20T08:47:23.586770362Z level=info msg="Executing migration" id="create dashboard v2" 2024-03-20T08:47:23.735 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Mar 20 08:47:23 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-grafana-a[51839]: logger=migrator t=2024-03-20T08:47:23.587576587Z level=info msg="Executing migration" id="create index IDX_dashboard_org_id - v2" 2024-03-20T08:47:23.735 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Mar 20 08:47:23 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-grafana-a[51839]: logger=migrator t=2024-03-20T08:47:23.588344576Z level=info msg="Executing migration" id="create index UQE_dashboard_org_id_slug - v2" 2024-03-20T08:47:23.735 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Mar 20 08:47:23 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-grafana-a[51839]: logger=migrator t=2024-03-20T08:47:23.589160672Z level=info msg="Executing migration" id="copy dashboard v1 to v2" 2024-03-20T08:47:23.735 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Mar 20 08:47:23 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-grafana-a[51839]: logger=migrator t=2024-03-20T08:47:23.589974085Z level=info msg="Executing migration" id="drop table dashboard_v1" 2024-03-20T08:47:23.736 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Mar 20 08:47:23 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-grafana-a[51839]: logger=migrator t=2024-03-20T08:47:23.590762138Z level=info msg="Executing migration" id="alter dashboard.data to mediumtext v1" 2024-03-20T08:47:23.736 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Mar 20 08:47:23 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-grafana-a[51839]: logger=migrator t=2024-03-20T08:47:23.591546917Z level=info msg="Executing migration" id="Add column updated_by in dashboard - v2" 2024-03-20T08:47:23.736 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Mar 20 08:47:23 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-grafana-a[51839]: logger=migrator t=2024-03-20T08:47:23.592328316Z level=info msg="Executing migration" id="Add column created_by in dashboard - v2" 2024-03-20T08:47:23.736 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Mar 20 08:47:23 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-grafana-a[51839]: logger=migrator t=2024-03-20T08:47:23.593153112Z level=info msg="Executing migration" id="Add column gnetId in dashboard" 2024-03-20T08:47:23.736 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Mar 20 08:47:23 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-grafana-a[51839]: logger=migrator t=2024-03-20T08:47:23.594035661Z level=info msg="Executing migration" id="Add index for gnetId in dashboard" 2024-03-20T08:47:23.736 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Mar 20 08:47:23 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-grafana-a[51839]: logger=migrator t=2024-03-20T08:47:23.594731783Z level=info msg="Executing migration" id="Add column plugin_id in dashboard" 2024-03-20T08:47:23.736 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Mar 20 08:47:23 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-grafana-a[51839]: logger=migrator t=2024-03-20T08:47:23.595516103Z level=info msg="Executing migration" id="Add index for plugin_id in dashboard" 2024-03-20T08:47:23.736 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Mar 20 08:47:23 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-grafana-a[51839]: logger=migrator t=2024-03-20T08:47:23.596304258Z level=info msg="Executing migration" id="Add index for dashboard_id in dashboard_tag" 2024-03-20T08:47:23.736 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Mar 20 08:47:23 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-grafana-a[51839]: logger=migrator t=2024-03-20T08:47:23.597120304Z level=info msg="Executing migration" id="Update dashboard table charset" 2024-03-20T08:47:23.736 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Mar 20 08:47:23 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-grafana-a[51839]: logger=migrator t=2024-03-20T08:47:23.598068414Z level=info msg="Executing migration" id="Update dashboard_tag table charset" 2024-03-20T08:47:23.736 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Mar 20 08:47:23 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-grafana-a[51839]: logger=migrator t=2024-03-20T08:47:23.598711013Z level=info msg="Executing migration" id="Add column folder_id in dashboard" 2024-03-20T08:47:23.736 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Mar 20 08:47:23 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-grafana-a[51839]: logger=migrator t=2024-03-20T08:47:23.599506314Z level=info msg="Executing migration" id="Add column isFolder in dashboard" 2024-03-20T08:47:23.737 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Mar 20 08:47:23 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-grafana-a[51839]: logger=migrator t=2024-03-20T08:47:23.600349587Z level=info msg="Executing migration" id="Add column has_acl in dashboard" 2024-03-20T08:47:23.737 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Mar 20 08:47:23 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-grafana-a[51839]: logger=migrator t=2024-03-20T08:47:23.6010632Z level=info msg="Executing migration" id="Add column uid in dashboard" 2024-03-20T08:47:23.737 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Mar 20 08:47:23 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-grafana-a[51839]: logger=migrator t=2024-03-20T08:47:23.603971135Z level=info msg="Executing migration" id="Update uid column values in dashboard" 2024-03-20T08:47:23.737 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Mar 20 08:47:23 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-grafana-a[51839]: logger=migrator t=2024-03-20T08:47:23.606501018Z level=info msg="Executing migration" id="Add unique index dashboard_org_id_uid" 2024-03-20T08:47:23.737 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Mar 20 08:47:23 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-grafana-a[51839]: logger=migrator t=2024-03-20T08:47:23.609226961Z level=info msg="Executing migration" id="Remove unique index org_id_slug" 2024-03-20T08:47:23.737 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Mar 20 08:47:23 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-grafana-a[51839]: logger=migrator t=2024-03-20T08:47:23.612012082Z level=info msg="Executing migration" id="Update dashboard title length" 2024-03-20T08:47:23.737 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Mar 20 08:47:23 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-grafana-a[51839]: logger=migrator t=2024-03-20T08:47:23.614735267Z level=info msg="Executing migration" id="Add unique index for dashboard_org_id_title_folder_id" 2024-03-20T08:47:23.737 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Mar 20 08:47:23 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-grafana-a[51839]: logger=migrator t=2024-03-20T08:47:23.617413928Z level=info msg="Executing migration" id="create dashboard_provisioning" 2024-03-20T08:47:23.737 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Mar 20 08:47:23 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-grafana-a[51839]: logger=migrator t=2024-03-20T08:47:23.619894079Z level=info msg="Executing migration" id="Rename table dashboard_provisioning to dashboard_provisioning_tmp_qwerty - v1" 2024-03-20T08:47:23.737 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Mar 20 08:47:23 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-grafana-a[51839]: logger=migrator t=2024-03-20T08:47:23.622812896Z level=info msg="Executing migration" id="create dashboard_provisioning v2" 2024-03-20T08:47:23.737 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Mar 20 08:47:23 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-grafana-a[51839]: logger=migrator t=2024-03-20T08:47:23.625489106Z level=info msg="Executing migration" id="create index IDX_dashboard_provisioning_dashboard_id - v2" 2024-03-20T08:47:23.738 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Mar 20 08:47:23 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-grafana-a[51839]: logger=migrator t=2024-03-20T08:47:23.628027675Z level=info msg="Executing migration" id="create index IDX_dashboard_provisioning_dashboard_id_name - v2" 2024-03-20T08:47:23.738 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Mar 20 08:47:23 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-grafana-a[51839]: logger=migrator t=2024-03-20T08:47:23.630852716Z level=info msg="Executing migration" id="copy dashboard_provisioning v1 to v2" 2024-03-20T08:47:23.738 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Mar 20 08:47:23 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-grafana-a[51839]: logger=migrator t=2024-03-20T08:47:23.63346603Z level=info msg="Executing migration" id="drop dashboard_provisioning_tmp_qwerty" 2024-03-20T08:47:23.738 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Mar 20 08:47:23 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-grafana-a[51839]: logger=migrator t=2024-03-20T08:47:23.636052905Z level=info msg="Executing migration" id="Add check_sum column" 2024-03-20T08:47:23.738 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Mar 20 08:47:23 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-grafana-a[51839]: logger=migrator t=2024-03-20T08:47:23.63870554Z level=info msg="Executing migration" id="Add index for dashboard_title" 2024-03-20T08:47:23.738 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Mar 20 08:47:23 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-grafana-a[51839]: logger=migrator t=2024-03-20T08:47:23.641450285Z level=info msg="Executing migration" id="delete tags for deleted dashboards" 2024-03-20T08:47:23.738 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Mar 20 08:47:23 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-grafana-a[51839]: logger=migrator t=2024-03-20T08:47:23.644418681Z level=info msg="Executing migration" id="delete stars for deleted dashboards" 2024-03-20T08:47:23.739 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Mar 20 08:47:23 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-grafana-a[51839]: logger=migrator t=2024-03-20T08:47:23.64711002Z level=info msg="Executing migration" id="Add index for dashboard_is_folder" 2024-03-20T08:47:23.739 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Mar 20 08:47:23 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-grafana-a[51839]: logger=migrator t=2024-03-20T08:47:23.649966335Z level=info msg="Executing migration" id="Add isPublic for dashboard" 2024-03-20T08:47:23.739 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Mar 20 08:47:23 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-grafana-a[51839]: logger=migrator t=2024-03-20T08:47:23.65256836Z level=info msg="Executing migration" id="create data_source table" 2024-03-20T08:47:23.739 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Mar 20 08:47:23 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-grafana-a[51839]: logger=migrator t=2024-03-20T08:47:23.65506623Z level=info msg="Executing migration" id="add index data_source.account_id" 2024-03-20T08:47:23.739 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Mar 20 08:47:23 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-grafana-a[51839]: logger=migrator t=2024-03-20T08:47:23.657594201Z level=info msg="Executing migration" id="add unique index data_source.account_id_name" 2024-03-20T08:47:23.739 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Mar 20 08:47:23 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-grafana-a[51839]: logger=migrator t=2024-03-20T08:47:23.660171885Z level=info msg="Executing migration" id="drop index IDX_data_source_account_id - v1" 2024-03-20T08:47:23.739 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Mar 20 08:47:23 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-grafana-a[51839]: logger=migrator t=2024-03-20T08:47:23.662405701Z level=info msg="Executing migration" id="drop index UQE_data_source_account_id_name - v1" 2024-03-20T08:47:23.739 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Mar 20 08:47:23 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-grafana-a[51839]: logger=migrator t=2024-03-20T08:47:23.664941702Z level=info msg="Executing migration" id="Rename table data_source to data_source_v1 - v1" 2024-03-20T08:47:23.739 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Mar 20 08:47:23 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-grafana-a[51839]: logger=migrator t=2024-03-20T08:47:23.667341583Z level=info msg="Executing migration" id="create data_source table v2" 2024-03-20T08:47:23.739 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Mar 20 08:47:23 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-grafana-a[51839]: logger=migrator t=2024-03-20T08:47:23.66999934Z level=info msg="Executing migration" id="create index IDX_data_source_org_id - v2" 2024-03-20T08:47:23.739 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Mar 20 08:47:23 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-grafana-a[51839]: logger=migrator t=2024-03-20T08:47:23.672406499Z level=info msg="Executing migration" id="create index UQE_data_source_org_id_name - v2" 2024-03-20T08:47:23.739 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Mar 20 08:47:23 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-grafana-a[51839]: logger=migrator t=2024-03-20T08:47:23.675079388Z level=info msg="Executing migration" id="Drop old table data_source_v1 #2" 2024-03-20T08:47:23.740 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Mar 20 08:47:23 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-grafana-a[51839]: logger=migrator t=2024-03-20T08:47:23.677537916Z level=info msg="Executing migration" id="Add column with_credentials" 2024-03-20T08:47:23.740 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Mar 20 08:47:23 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-grafana-a[51839]: logger=migrator t=2024-03-20T08:47:23.680225878Z level=info msg="Executing migration" id="Add secure json data column" 2024-03-20T08:47:23.740 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Mar 20 08:47:23 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-grafana-a[51839]: logger=migrator t=2024-03-20T08:47:23.68254306Z level=info msg="Executing migration" id="Update data_source table charset" 2024-03-20T08:47:23.740 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Mar 20 08:47:23 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-grafana-a[51839]: logger=migrator t=2024-03-20T08:47:23.684934523Z level=info msg="Executing migration" id="Update initial version to 1" 2024-03-20T08:47:23.740 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Mar 20 08:47:23 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-grafana-a[51839]: logger=migrator t=2024-03-20T08:47:23.687625227Z level=info msg="Executing migration" id="Add read_only data column" 2024-03-20T08:47:23.740 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Mar 20 08:47:23 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-grafana-a[51839]: logger=migrator t=2024-03-20T08:47:23.690296853Z level=info msg="Executing migration" id="Migrate logging ds to loki ds" 2024-03-20T08:47:23.740 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Mar 20 08:47:23 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-grafana-a[51839]: logger=migrator t=2024-03-20T08:47:23.692873792Z level=info msg="Executing migration" id="Update json_data with nulls" 2024-03-20T08:47:23.740 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Mar 20 08:47:23 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-grafana-a[51839]: logger=migrator t=2024-03-20T08:47:23.693572787Z level=info msg="Executing migration" id="Add uid column" 2024-03-20T08:47:23.740 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Mar 20 08:47:23 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-grafana-a[51839]: logger=migrator t=2024-03-20T08:47:23.694327929Z level=info msg="Executing migration" id="Update uid value" 2024-03-20T08:47:23.740 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Mar 20 08:47:23 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-grafana-a[51839]: logger=migrator t=2024-03-20T08:47:23.695106923Z level=info msg="Executing migration" id="Add unique index datasource_org_id_uid" 2024-03-20T08:47:23.740 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Mar 20 08:47:23 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-grafana-a[51839]: logger=migrator t=2024-03-20T08:47:23.695824584Z level=info msg="Executing migration" id="add unique index datasource_org_id_is_default" 2024-03-20T08:47:23.740 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Mar 20 08:47:23 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-grafana-a[51839]: logger=migrator t=2024-03-20T08:47:23.69656572Z level=info msg="Executing migration" id="create api_key table" 2024-03-20T08:47:23.741 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Mar 20 08:47:23 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-grafana-a[51839]: logger=migrator t=2024-03-20T08:47:23.697259419Z level=info msg="Executing migration" id="add index api_key.account_id" 2024-03-20T08:47:23.741 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Mar 20 08:47:23 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-grafana-a[51839]: logger=migrator t=2024-03-20T08:47:23.698017437Z level=info msg="Executing migration" id="add index api_key.key" 2024-03-20T08:47:23.741 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Mar 20 08:47:23 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-grafana-a[51839]: logger=migrator t=2024-03-20T08:47:23.698848918Z level=info msg="Executing migration" id="add index api_key.account_id_name" 2024-03-20T08:47:23.741 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Mar 20 08:47:23 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-grafana-a[51839]: logger=migrator t=2024-03-20T08:47:23.699454522Z level=info msg="Executing migration" id="drop index IDX_api_key_account_id - v1" 2024-03-20T08:47:23.741 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Mar 20 08:47:23 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-grafana-a[51839]: logger=migrator t=2024-03-20T08:47:23.700138136Z level=info msg="Executing migration" id="drop index UQE_api_key_key - v1" 2024-03-20T08:47:23.741 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Mar 20 08:47:23 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-grafana-a[51839]: logger=migrator t=2024-03-20T08:47:23.700523657Z level=info msg="Executing migration" id="drop index UQE_api_key_account_id_name - v1" 2024-03-20T08:47:23.741 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Mar 20 08:47:23 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-grafana-a[51839]: logger=migrator t=2024-03-20T08:47:23.700885446Z level=info msg="Executing migration" id="Rename table api_key to api_key_v1 - v1" 2024-03-20T08:47:23.741 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Mar 20 08:47:23 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-grafana-a[51839]: logger=migrator t=2024-03-20T08:47:23.702017605Z level=info msg="Executing migration" id="create api_key table v2" 2024-03-20T08:47:23.741 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Mar 20 08:47:23 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-grafana-a[51839]: logger=migrator t=2024-03-20T08:47:23.7023978Z level=info msg="Executing migration" id="create index IDX_api_key_org_id - v2" 2024-03-20T08:47:23.741 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Mar 20 08:47:23 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-grafana-a[51839]: logger=migrator t=2024-03-20T08:47:23.70277624Z level=info msg="Executing migration" id="create index UQE_api_key_key - v2" 2024-03-20T08:47:23.741 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Mar 20 08:47:23 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-grafana-a[51839]: logger=migrator t=2024-03-20T08:47:23.703181611Z level=info msg="Executing migration" id="create index UQE_api_key_org_id_name - v2" 2024-03-20T08:47:23.742 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Mar 20 08:47:23 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-grafana-a[51839]: logger=migrator t=2024-03-20T08:47:23.703649723Z level=info msg="Executing migration" id="copy api_key v1 to v2" 2024-03-20T08:47:23.742 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Mar 20 08:47:23 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-grafana-a[51839]: logger=migrator t=2024-03-20T08:47:23.704396369Z level=info msg="Executing migration" id="Drop old table api_key_v1" 2024-03-20T08:47:23.742 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Mar 20 08:47:23 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-grafana-a[51839]: logger=migrator t=2024-03-20T08:47:23.704752781Z level=info msg="Executing migration" id="Update api_key table charset" 2024-03-20T08:47:23.742 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Mar 20 08:47:23 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-grafana-a[51839]: logger=migrator t=2024-03-20T08:47:23.705194656Z level=info msg="Executing migration" id="Add expires to api_key table" 2024-03-20T08:47:23.742 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Mar 20 08:47:23 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-grafana-a[51839]: logger=migrator t=2024-03-20T08:47:23.705717635Z level=info msg="Executing migration" id="Add service account foreign key" 2024-03-20T08:47:23.742 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Mar 20 08:47:23 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-grafana-a[51839]: logger=migrator t=2024-03-20T08:47:23.70625031Z level=info msg="Executing migration" id="set service account foreign key to nil if 0" 2024-03-20T08:47:23.742 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Mar 20 08:47:23 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-grafana-a[51839]: logger=migrator t=2024-03-20T08:47:23.707127708Z level=info msg="Executing migration" id="Add last_used_at to api_key table" 2024-03-20T08:47:23.742 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Mar 20 08:47:23 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-grafana-a[51839]: logger=migrator t=2024-03-20T08:47:23.707813294Z level=info msg="Executing migration" id="Add is_revoked column to api_key table" 2024-03-20T08:47:23.742 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Mar 20 08:47:23 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-grafana-a[51839]: logger=migrator t=2024-03-20T08:47:23.708654403Z level=info msg="Executing migration" id="create dashboard_snapshot table v4" 2024-03-20T08:47:23.742 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Mar 20 08:47:23 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-grafana-a[51839]: logger=migrator t=2024-03-20T08:47:23.709557716Z level=info msg="Executing migration" id="drop table dashboard_snapshot_v4 #1" 2024-03-20T08:47:23.743 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Mar 20 08:47:23 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-grafana-a[51839]: logger=migrator t=2024-03-20T08:47:23.710260581Z level=info msg="Executing migration" id="create dashboard_snapshot table v5 #2" 2024-03-20T08:47:23.743 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Mar 20 08:47:23 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-grafana-a[51839]: logger=migrator t=2024-03-20T08:47:23.711132048Z level=info msg="Executing migration" id="create index UQE_dashboard_snapshot_key - v5" 2024-03-20T08:47:23.743 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Mar 20 08:47:23 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-grafana-a[51839]: logger=migrator t=2024-03-20T08:47:23.711812002Z level=info msg="Executing migration" id="create index UQE_dashboard_snapshot_delete_key - v5" 2024-03-20T08:47:23.743 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Mar 20 08:47:23 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-grafana-a[51839]: logger=migrator t=2024-03-20T08:47:23.712772362Z level=info msg="Executing migration" id="create index IDX_dashboard_snapshot_user_id - v5" 2024-03-20T08:47:23.743 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Mar 20 08:47:23 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-grafana-a[51839]: logger=migrator t=2024-03-20T08:47:23.713385999Z level=info msg="Executing migration" id="alter dashboard_snapshot to mediumtext v2" 2024-03-20T08:47:23.743 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Mar 20 08:47:23 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-grafana-a[51839]: logger=migrator t=2024-03-20T08:47:23.714162534Z level=info msg="Executing migration" id="Update dashboard_snapshot table charset" 2024-03-20T08:47:23.743 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Mar 20 08:47:23 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-grafana-a[51839]: logger=migrator t=2024-03-20T08:47:23.715011349Z level=info msg="Executing migration" id="Add column external_delete_url to dashboard_snapshots table" 2024-03-20T08:47:23.743 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Mar 20 08:47:23 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-grafana-a[51839]: logger=migrator t=2024-03-20T08:47:23.715814273Z level=info msg="Executing migration" id="Add encrypted dashboard json column" 2024-03-20T08:47:23.743 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Mar 20 08:47:23 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-grafana-a[51839]: logger=migrator t=2024-03-20T08:47:23.716563827Z level=info msg="Executing migration" id="Change dashboard_encrypted column to MEDIUMBLOB" 2024-03-20T08:47:23.743 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Mar 20 08:47:23 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-grafana-a[51839]: logger=migrator t=2024-03-20T08:47:23.719311287Z level=info msg="Executing migration" id="create quota table v1" 2024-03-20T08:47:23.743 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Mar 20 08:47:23 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-grafana-a[51839]: logger=migrator t=2024-03-20T08:47:23.72216378Z level=info msg="Executing migration" id="create index UQE_quota_org_id_user_id_target - v1" 2024-03-20T08:47:23.743 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Mar 20 08:47:23 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-grafana-a[51839]: logger=migrator t=2024-03-20T08:47:23.724997772Z level=info msg="Executing migration" id="Update quota table charset" 2024-03-20T08:47:23.744 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Mar 20 08:47:23 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-grafana-a[51839]: logger=migrator t=2024-03-20T08:47:23.727577402Z level=info msg="Executing migration" id="create plugin_setting table" 2024-03-20T08:47:23.977 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Mar 20 08:47:23 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-grafana-a[51839]: logger=migrator t=2024-03-20T08:47:23.730158212Z level=info msg="Executing migration" id="create index UQE_plugin_setting_org_id_plugin_id - v1" 2024-03-20T08:47:23.978 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Mar 20 08:47:23 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-grafana-a[51839]: logger=migrator t=2024-03-20T08:47:23.732934117Z level=info msg="Executing migration" id="Add column plugin_version to plugin_settings" 2024-03-20T08:47:23.978 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Mar 20 08:47:23 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-grafana-a[51839]: logger=migrator t=2024-03-20T08:47:23.735713211Z level=info msg="Executing migration" id="Update plugin_setting table charset" 2024-03-20T08:47:23.978 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Mar 20 08:47:23 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-grafana-a[51839]: logger=migrator t=2024-03-20T08:47:23.738444816Z level=info msg="Executing migration" id="create session table" 2024-03-20T08:47:23.978 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Mar 20 08:47:23 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-grafana-a[51839]: logger=migrator t=2024-03-20T08:47:23.740870506Z level=info msg="Executing migration" id="Drop old table playlist table" 2024-03-20T08:47:23.978 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Mar 20 08:47:23 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-grafana-a[51839]: logger=migrator t=2024-03-20T08:47:23.743594406Z level=info msg="Executing migration" id="Drop old table playlist_item table" 2024-03-20T08:47:23.978 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Mar 20 08:47:23 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-grafana-a[51839]: logger=migrator t=2024-03-20T08:47:23.746432082Z level=info msg="Executing migration" id="create playlist table v2" 2024-03-20T08:47:23.978 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Mar 20 08:47:23 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-grafana-a[51839]: logger=migrator t=2024-03-20T08:47:23.749199708Z level=info msg="Executing migration" id="create playlist item table v2" 2024-03-20T08:47:23.978 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Mar 20 08:47:23 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-grafana-a[51839]: logger=migrator t=2024-03-20T08:47:23.751956463Z level=info msg="Executing migration" id="Update playlist table charset" 2024-03-20T08:47:23.978 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Mar 20 08:47:23 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-grafana-a[51839]: logger=migrator t=2024-03-20T08:47:23.754736356Z level=info msg="Executing migration" id="Update playlist_item table charset" 2024-03-20T08:47:23.978 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Mar 20 08:47:23 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-grafana-a[51839]: logger=migrator t=2024-03-20T08:47:23.757480141Z level=info msg="Executing migration" id="drop preferences table v2" 2024-03-20T08:47:23.978 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Mar 20 08:47:23 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-grafana-a[51839]: logger=migrator t=2024-03-20T08:47:23.760316964Z level=info msg="Executing migration" id="drop preferences table v3" 2024-03-20T08:47:23.978 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Mar 20 08:47:23 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-grafana-a[51839]: logger=migrator t=2024-03-20T08:47:23.763038811Z level=info msg="Executing migration" id="create preferences table v3" 2024-03-20T08:47:23.978 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Mar 20 08:47:23 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-grafana-a[51839]: logger=migrator t=2024-03-20T08:47:23.765706189Z level=info msg="Executing migration" id="Update preferences table charset" 2024-03-20T08:47:23.979 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Mar 20 08:47:23 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-grafana-a[51839]: logger=migrator t=2024-03-20T08:47:23.768332733Z level=info msg="Executing migration" id="Add column team_id in preferences" 2024-03-20T08:47:23.979 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Mar 20 08:47:23 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-grafana-a[51839]: logger=migrator t=2024-03-20T08:47:23.770825147Z level=info msg="Executing migration" id="Update team_id column values in preferences" 2024-03-20T08:47:23.979 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Mar 20 08:47:23 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-grafana-a[51839]: logger=migrator t=2024-03-20T08:47:23.773079753Z level=info msg="Executing migration" id="Add column week_start in preferences" 2024-03-20T08:47:23.979 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Mar 20 08:47:23 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-grafana-a[51839]: logger=migrator t=2024-03-20T08:47:23.775618753Z level=info msg="Executing migration" id="Add column preferences.json_data" 2024-03-20T08:47:23.979 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Mar 20 08:47:23 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-grafana-a[51839]: logger=migrator t=2024-03-20T08:47:23.778023499Z level=info msg="Executing migration" id="alter preferences.json_data to mediumtext v1" 2024-03-20T08:47:23.979 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Mar 20 08:47:23 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-grafana-a[51839]: logger=migrator t=2024-03-20T08:47:23.780279745Z level=info msg="Executing migration" id="Add preferences index org_id" 2024-03-20T08:47:23.979 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Mar 20 08:47:23 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-grafana-a[51839]: logger=migrator t=2024-03-20T08:47:23.783053367Z level=info msg="Executing migration" id="Add preferences index user_id" 2024-03-20T08:47:23.979 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Mar 20 08:47:23 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-grafana-a[51839]: logger=migrator t=2024-03-20T08:47:23.785511078Z level=info msg="Executing migration" id="create alert table v1" 2024-03-20T08:47:23.979 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Mar 20 08:47:23 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-grafana-a[51839]: logger=migrator t=2024-03-20T08:47:23.788119796Z level=info msg="Executing migration" id="add index alert org_id & id " 2024-03-20T08:47:23.979 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Mar 20 08:47:23 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-grafana-a[51839]: logger=migrator t=2024-03-20T08:47:23.790671898Z level=info msg="Executing migration" id="add index alert state" 2024-03-20T08:47:23.979 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Mar 20 08:47:23 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-grafana-a[51839]: logger=migrator t=2024-03-20T08:47:23.793288965Z level=info msg="Executing migration" id="add index alert dashboard_id" 2024-03-20T08:47:23.980 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Mar 20 08:47:23 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-grafana-a[51839]: logger=migrator t=2024-03-20T08:47:23.795838941Z level=info msg="Executing migration" id="Create alert_rule_tag table v1" 2024-03-20T08:47:23.980 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Mar 20 08:47:23 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-grafana-a[51839]: logger=migrator t=2024-03-20T08:47:23.796606075Z level=info msg="Executing migration" id="Add unique index alert_rule_tag.alert_id_tag_id" 2024-03-20T08:47:23.980 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Mar 20 08:47:23 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-grafana-a[51839]: logger=migrator t=2024-03-20T08:47:23.799493117Z level=info msg="Executing migration" id="drop index UQE_alert_rule_tag_alert_id_tag_id - v1" 2024-03-20T08:47:23.980 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Mar 20 08:47:23 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-grafana-a[51839]: logger=migrator t=2024-03-20T08:47:23.802167323Z level=info msg="Executing migration" id="Rename table alert_rule_tag to alert_rule_tag_v1 - v1" 2024-03-20T08:47:23.980 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Mar 20 08:47:23 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-grafana-a[51839]: logger=migrator t=2024-03-20T08:47:23.804957679Z level=info msg="Executing migration" id="Create alert_rule_tag table v2" 2024-03-20T08:47:23.980 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Mar 20 08:47:23 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-grafana-a[51839]: logger=migrator t=2024-03-20T08:47:23.807810251Z level=info msg="Executing migration" id="create index UQE_alert_rule_tag_alert_id_tag_id - Add unique index alert_rule_tag.alert_id_tag_id V2" 2024-03-20T08:47:23.980 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Mar 20 08:47:23 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-grafana-a[51839]: logger=migrator t=2024-03-20T08:47:23.81049501Z level=info msg="Executing migration" id="copy alert_rule_tag v1 to v2" 2024-03-20T08:47:23.980 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Mar 20 08:47:23 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-grafana-a[51839]: logger=migrator t=2024-03-20T08:47:23.813026147Z level=info msg="Executing migration" id="drop table alert_rule_tag_v1" 2024-03-20T08:47:23.980 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Mar 20 08:47:23 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-grafana-a[51839]: logger=migrator t=2024-03-20T08:47:23.813428167Z level=info msg="Executing migration" id="create alert_notification table v1" 2024-03-20T08:47:23.980 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Mar 20 08:47:23 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-grafana-a[51839]: logger=migrator t=2024-03-20T08:47:23.816018616Z level=info msg="Executing migration" id="Add column is_default" 2024-03-20T08:47:23.980 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Mar 20 08:47:23 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-grafana-a[51839]: logger=migrator t=2024-03-20T08:47:23.816656428Z level=info msg="Executing migration" id="Add column frequency" 2024-03-20T08:47:23.980 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Mar 20 08:47:23 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-grafana-a[51839]: logger=migrator t=2024-03-20T08:47:23.818907489Z level=info msg="Executing migration" id="Add column send_reminder" 2024-03-20T08:47:23.980 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Mar 20 08:47:23 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-grafana-a[51839]: logger=migrator t=2024-03-20T08:47:23.821780067Z level=info msg="Executing migration" id="Add column disable_resolve_message" 2024-03-20T08:47:23.981 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Mar 20 08:47:23 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-grafana-a[51839]: logger=migrator t=2024-03-20T08:47:23.824249802Z level=info msg="Executing migration" id="add index alert_notification org_id & name" 2024-03-20T08:47:23.981 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Mar 20 08:47:23 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-grafana-a[51839]: logger=migrator t=2024-03-20T08:47:23.827162408Z level=info msg="Executing migration" id="Update alert table charset" 2024-03-20T08:47:23.981 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Mar 20 08:47:23 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-grafana-a[51839]: logger=migrator t=2024-03-20T08:47:23.829711515Z level=info msg="Executing migration" id="Update alert_notification table charset" 2024-03-20T08:47:23.981 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Mar 20 08:47:23 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-grafana-a[51839]: logger=migrator t=2024-03-20T08:47:23.830021477Z level=info msg="Executing migration" id="create notification_journal table v1" 2024-03-20T08:47:23.981 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Mar 20 08:47:23 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-grafana-a[51839]: logger=migrator t=2024-03-20T08:47:23.830421842Z level=info msg="Executing migration" id="add index notification_journal org_id & alert_id & notifier_id" 2024-03-20T08:47:23.981 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Mar 20 08:47:23 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-grafana-a[51839]: logger=migrator t=2024-03-20T08:47:23.833303145Z level=info msg="Executing migration" id="drop alert_notification_journal" 2024-03-20T08:47:23.981 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Mar 20 08:47:23 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-grafana-a[51839]: logger=migrator t=2024-03-20T08:47:23.835852043Z level=info msg="Executing migration" id="create alert_notification_state table v1" 2024-03-20T08:47:23.981 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Mar 20 08:47:23 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-grafana-a[51839]: logger=migrator t=2024-03-20T08:47:23.838682937Z level=info msg="Executing migration" id="add index alert_notification_state org_id & alert_id & notifier_id" 2024-03-20T08:47:23.981 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Mar 20 08:47:23 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-grafana-a[51839]: logger=migrator t=2024-03-20T08:47:23.841334431Z level=info msg="Executing migration" id="Add for to alert table" 2024-03-20T08:47:23.981 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Mar 20 08:47:23 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-grafana-a[51839]: logger=migrator t=2024-03-20T08:47:23.843905871Z level=info msg="Executing migration" id="Add column uid in alert_notification" 2024-03-20T08:47:23.981 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Mar 20 08:47:23 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-grafana-a[51839]: logger=migrator t=2024-03-20T08:47:23.845069629Z level=info msg="Executing migration" id="Update uid column values in alert_notification" 2024-03-20T08:47:23.981 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Mar 20 08:47:23 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-grafana-a[51839]: logger=migrator t=2024-03-20T08:47:23.845634108Z level=info msg="Executing migration" id="Add unique index alert_notification_org_id_uid" 2024-03-20T08:47:23.981 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Mar 20 08:47:23 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-grafana-a[51839]: logger=migrator t=2024-03-20T08:47:23.846495848Z level=info msg="Executing migration" id="Remove unique index org_id_name" 2024-03-20T08:47:23.981 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Mar 20 08:47:23 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-grafana-a[51839]: logger=migrator t=2024-03-20T08:47:23.847341808Z level=info msg="Executing migration" id="Add column secure_settings in alert_notification" 2024-03-20T08:47:23.981 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Mar 20 08:47:23 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-grafana-a[51839]: logger=migrator t=2024-03-20T08:47:23.848195794Z level=info msg="Executing migration" id="alter alert.settings to mediumtext" 2024-03-20T08:47:23.982 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Mar 20 08:47:23 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-grafana-a[51839]: logger=migrator t=2024-03-20T08:47:23.849120624Z level=info msg="Executing migration" id="Add non-unique index alert_notification_state_alert_id" 2024-03-20T08:47:23.982 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Mar 20 08:47:23 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-grafana-a[51839]: logger=migrator t=2024-03-20T08:47:23.849865853Z level=info msg="Executing migration" id="Add non-unique index alert_rule_tag_alert_id" 2024-03-20T08:47:23.982 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Mar 20 08:47:23 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-grafana-a[51839]: logger=migrator t=2024-03-20T08:47:23.850839423Z level=info msg="Executing migration" id="Drop old annotation table v4" 2024-03-20T08:47:23.982 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Mar 20 08:47:23 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-grafana-a[51839]: logger=migrator t=2024-03-20T08:47:23.851600093Z level=info msg="Executing migration" id="create annotation table v5" 2024-03-20T08:47:23.983 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Mar 20 08:47:23 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-grafana-a[51839]: logger=migrator t=2024-03-20T08:47:23.852412815Z level=info msg="Executing migration" id="add index annotation 0 v3" 2024-03-20T08:47:23.983 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Mar 20 08:47:23 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-grafana-a[51839]: logger=migrator t=2024-03-20T08:47:23.853278774Z level=info msg="Executing migration" id="add index annotation 1 v3" 2024-03-20T08:47:23.983 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Mar 20 08:47:23 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-grafana-a[51839]: logger=migrator t=2024-03-20T08:47:23.854255685Z level=info msg="Executing migration" id="add index annotation 2 v3" 2024-03-20T08:47:23.983 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Mar 20 08:47:23 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-grafana-a[51839]: logger=migrator t=2024-03-20T08:47:23.855003266Z level=info msg="Executing migration" id="add index annotation 3 v3" 2024-03-20T08:47:23.983 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Mar 20 08:47:23 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-grafana-a[51839]: logger=migrator t=2024-03-20T08:47:23.855847099Z level=info msg="Executing migration" id="add index annotation 4 v3" 2024-03-20T08:47:23.983 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Mar 20 08:47:23 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-grafana-a[51839]: logger=migrator t=2024-03-20T08:47:23.856729033Z level=info msg="Executing migration" id="Update annotation table charset" 2024-03-20T08:47:23.983 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Mar 20 08:47:23 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-grafana-a[51839]: logger=migrator t=2024-03-20T08:47:23.857538233Z level=info msg="Executing migration" id="Add column region_id to annotation table" 2024-03-20T08:47:23.983 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Mar 20 08:47:23 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-grafana-a[51839]: logger=migrator t=2024-03-20T08:47:23.858455209Z level=info msg="Executing migration" id="Drop category_id index" 2024-03-20T08:47:23.983 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Mar 20 08:47:23 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-grafana-a[51839]: logger=migrator t=2024-03-20T08:47:23.859287868Z level=info msg="Executing migration" id="Add column tags to annotation table" 2024-03-20T08:47:23.983 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Mar 20 08:47:23 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-grafana-a[51839]: logger=migrator t=2024-03-20T08:47:23.860140065Z level=info msg="Executing migration" id="Create annotation_tag table v2" 2024-03-20T08:47:23.984 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Mar 20 08:47:23 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-grafana-a[51839]: logger=migrator t=2024-03-20T08:47:23.861141256Z level=info msg="Executing migration" id="Add unique index annotation_tag.annotation_id_tag_id" 2024-03-20T08:47:23.984 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Mar 20 08:47:23 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-grafana-a[51839]: logger=migrator t=2024-03-20T08:47:23.861853167Z level=info msg="Executing migration" id="drop index UQE_annotation_tag_annotation_id_tag_id - v2" 2024-03-20T08:47:23.984 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Mar 20 08:47:23 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-grafana-a[51839]: logger=migrator t=2024-03-20T08:47:23.86270058Z level=info msg="Executing migration" id="Rename table annotation_tag to annotation_tag_v2 - v2" 2024-03-20T08:47:23.984 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Mar 20 08:47:23 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-grafana-a[51839]: logger=migrator t=2024-03-20T08:47:23.86431845Z level=info msg="Executing migration" id="Create annotation_tag table v3" 2024-03-20T08:47:23.984 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Mar 20 08:47:23 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-grafana-a[51839]: logger=migrator t=2024-03-20T08:47:23.865273943Z level=info msg="Executing migration" id="create index UQE_annotation_tag_annotation_id_tag_id - Add unique index annotation_tag.annotation_id_tag_id V3" 2024-03-20T08:47:23.984 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Mar 20 08:47:23 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-grafana-a[51839]: logger=migrator t=2024-03-20T08:47:23.866058285Z level=info msg="Executing migration" id="copy annotation_tag v2 to v3" 2024-03-20T08:47:23.984 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Mar 20 08:47:23 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-grafana-a[51839]: logger=migrator t=2024-03-20T08:47:23.867040572Z level=info msg="Executing migration" id="drop table annotation_tag_v2" 2024-03-20T08:47:23.984 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Mar 20 08:47:23 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-grafana-a[51839]: logger=migrator t=2024-03-20T08:47:23.867795229Z level=info msg="Executing migration" id="Update alert annotations and set TEXT to empty" 2024-03-20T08:47:23.984 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Mar 20 08:47:23 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-grafana-a[51839]: logger=migrator t=2024-03-20T08:47:23.86876209Z level=info msg="Executing migration" id="Add created time to annotation table" 2024-03-20T08:47:23.985 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Mar 20 08:47:23 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-grafana-a[51839]: logger=migrator t=2024-03-20T08:47:23.869536054Z level=info msg="Executing migration" id="Add updated time to annotation table" 2024-03-20T08:47:23.985 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Mar 20 08:47:23 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-grafana-a[51839]: logger=migrator t=2024-03-20T08:47:23.870398617Z level=info msg="Executing migration" id="Add index for created in annotation table" 2024-03-20T08:47:23.985 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Mar 20 08:47:23 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-grafana-a[51839]: logger=migrator t=2024-03-20T08:47:23.871346791Z level=info msg="Executing migration" id="Add index for updated in annotation table" 2024-03-20T08:47:23.985 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Mar 20 08:47:23 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-grafana-a[51839]: logger=migrator t=2024-03-20T08:47:23.872088512Z level=info msg="Executing migration" id="Convert existing annotations from seconds to milliseconds" 2024-03-20T08:47:23.985 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Mar 20 08:47:23 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-grafana-a[51839]: logger=migrator t=2024-03-20T08:47:23.872996933Z level=info msg="Executing migration" id="Add epoch_end column" 2024-03-20T08:47:23.985 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Mar 20 08:47:23 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-grafana-a[51839]: logger=migrator t=2024-03-20T08:47:23.873879621Z level=info msg="Executing migration" id="Add index for epoch_end" 2024-03-20T08:47:23.985 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Mar 20 08:47:23 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-grafana-a[51839]: logger=migrator t=2024-03-20T08:47:23.874741672Z level=info msg="Executing migration" id="Make epoch_end the same as epoch" 2024-03-20T08:47:23.985 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Mar 20 08:47:23 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-grafana-a[51839]: logger=migrator t=2024-03-20T08:47:23.878435947Z level=info msg="Executing migration" id="Move region to single row" 2024-03-20T08:47:23.985 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Mar 20 08:47:23 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-grafana-a[51839]: logger=migrator t=2024-03-20T08:47:23.880369228Z level=info msg="Executing migration" id="Remove index org_id_epoch from annotation table" 2024-03-20T08:47:23.985 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Mar 20 08:47:23 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-grafana-a[51839]: logger=migrator t=2024-03-20T08:47:23.883445207Z level=info msg="Executing migration" id="Remove index org_id_dashboard_id_panel_id_epoch from annotation table" 2024-03-20T08:47:23.985 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Mar 20 08:47:23 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-grafana-a[51839]: logger=migrator t=2024-03-20T08:47:23.886121138Z level=info msg="Executing migration" id="Add index for org_id_dashboard_id_epoch_end_epoch on annotation table" 2024-03-20T08:47:23.985 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Mar 20 08:47:23 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-grafana-a[51839]: logger=migrator t=2024-03-20T08:47:23.888977322Z level=info msg="Executing migration" id="Add index for org_id_epoch_end_epoch on annotation table" 2024-03-20T08:47:23.985 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Mar 20 08:47:23 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-grafana-a[51839]: logger=migrator t=2024-03-20T08:47:23.891974137Z level=info msg="Executing migration" id="Remove index org_id_epoch_epoch_end from annotation table" 2024-03-20T08:47:23.985 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Mar 20 08:47:23 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-grafana-a[51839]: logger=migrator t=2024-03-20T08:47:23.894622473Z level=info msg="Executing migration" id="Add index for alert_id on annotation table" 2024-03-20T08:47:23.985 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Mar 20 08:47:23 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-grafana-a[51839]: logger=migrator t=2024-03-20T08:47:23.897208852Z level=info msg="Executing migration" id="Increase tags column to length 4096" 2024-03-20T08:47:23.986 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Mar 20 08:47:23 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-grafana-a[51839]: logger=migrator t=2024-03-20T08:47:23.900137801Z level=info msg="Executing migration" id="create test_data table" 2024-03-20T08:47:23.986 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Mar 20 08:47:23 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-grafana-a[51839]: logger=migrator t=2024-03-20T08:47:23.902985303Z level=info msg="Executing migration" id="create dashboard_version table v1" 2024-03-20T08:47:23.986 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Mar 20 08:47:23 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-grafana-a[51839]: logger=migrator t=2024-03-20T08:47:23.903420737Z level=info msg="Executing migration" id="add index dashboard_version.dashboard_id" 2024-03-20T08:47:23.986 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Mar 20 08:47:23 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-grafana-a[51839]: logger=migrator t=2024-03-20T08:47:23.906202105Z level=info msg="Executing migration" id="add unique index dashboard_version.dashboard_id and dashboard_version.version" 2024-03-20T08:47:23.986 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Mar 20 08:47:23 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-grafana-a[51839]: logger=migrator t=2024-03-20T08:47:23.908957132Z level=info msg="Executing migration" id="Set dashboard version to 1 where 0" 2024-03-20T08:47:23.986 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Mar 20 08:47:23 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-grafana-a[51839]: logger=migrator t=2024-03-20T08:47:23.911765846Z level=info msg="Executing migration" id="save existing dashboard data in dashboard_version table v1" 2024-03-20T08:47:23.986 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Mar 20 08:47:23 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-grafana-a[51839]: logger=migrator t=2024-03-20T08:47:23.914554632Z level=info msg="Executing migration" id="alter dashboard_version.data to mediumtext v1" 2024-03-20T08:47:23.986 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Mar 20 08:47:23 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-grafana-a[51839]: logger=migrator t=2024-03-20T08:47:23.917423342Z level=info msg="Executing migration" id="create team table" 2024-03-20T08:47:23.986 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Mar 20 08:47:23 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-grafana-a[51839]: logger=migrator t=2024-03-20T08:47:23.920415266Z level=info msg="Executing migration" id="add index team.org_id" 2024-03-20T08:47:23.986 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Mar 20 08:47:23 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-grafana-a[51839]: logger=migrator t=2024-03-20T08:47:23.923095998Z level=info msg="Executing migration" id="add unique index team_org_id_name" 2024-03-20T08:47:23.986 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Mar 20 08:47:23 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-grafana-a[51839]: logger=migrator t=2024-03-20T08:47:23.925988132Z level=info msg="Executing migration" id="create team member table" 2024-03-20T08:47:23.986 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Mar 20 08:47:23 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-grafana-a[51839]: logger=migrator t=2024-03-20T08:47:23.928720085Z level=info msg="Executing migration" id="add index team_member.org_id" 2024-03-20T08:47:23.986 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Mar 20 08:47:23 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-grafana-a[51839]: logger=migrator t=2024-03-20T08:47:23.931358605Z level=info msg="Executing migration" id="add unique index team_member_org_id_team_id_user_id" 2024-03-20T08:47:23.987 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Mar 20 08:47:23 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-grafana-a[51839]: logger=migrator t=2024-03-20T08:47:23.934100415Z level=info msg="Executing migration" id="add index team_member.team_id" 2024-03-20T08:47:23.987 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Mar 20 08:47:23 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-grafana-a[51839]: logger=migrator t=2024-03-20T08:47:23.936747915Z level=info msg="Executing migration" id="Add column email to team table" 2024-03-20T08:47:23.987 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Mar 20 08:47:23 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-grafana-a[51839]: logger=migrator t=2024-03-20T08:47:23.939570529Z level=info msg="Executing migration" id="Add column external to team_member table" 2024-03-20T08:47:23.987 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Mar 20 08:47:23 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-grafana-a[51839]: logger=migrator t=2024-03-20T08:47:23.942316604Z level=info msg="Executing migration" id="Add column permission to team_member table" 2024-03-20T08:47:23.987 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Mar 20 08:47:23 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-grafana-a[51839]: logger=migrator t=2024-03-20T08:47:23.945150053Z level=info msg="Executing migration" id="create dashboard acl table" 2024-03-20T08:47:23.987 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Mar 20 08:47:23 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-grafana-a[51839]: logger=migrator t=2024-03-20T08:47:23.947864269Z level=info msg="Executing migration" id="add index dashboard_acl_dashboard_id" 2024-03-20T08:47:23.987 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Mar 20 08:47:23 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-grafana-a[51839]: logger=migrator t=2024-03-20T08:47:23.950709565Z level=info msg="Executing migration" id="add unique index dashboard_acl_dashboard_id_user_id" 2024-03-20T08:47:23.987 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Mar 20 08:47:23 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-grafana-a[51839]: logger=migrator t=2024-03-20T08:47:23.953324546Z level=info msg="Executing migration" id="add unique index dashboard_acl_dashboard_id_team_id" 2024-03-20T08:47:23.987 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Mar 20 08:47:23 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-grafana-a[51839]: logger=migrator t=2024-03-20T08:47:23.95395558Z level=info msg="Executing migration" id="add index dashboard_acl_user_id" 2024-03-20T08:47:23.987 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Mar 20 08:47:23 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-grafana-a[51839]: logger=migrator t=2024-03-20T08:47:23.956889778Z level=info msg="Executing migration" id="add index dashboard_acl_team_id" 2024-03-20T08:47:23.987 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Mar 20 08:47:23 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-grafana-a[51839]: logger=migrator t=2024-03-20T08:47:23.957358561Z level=info msg="Executing migration" id="add index dashboard_acl_org_id_role" 2024-03-20T08:47:23.987 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Mar 20 08:47:23 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-grafana-a[51839]: logger=migrator t=2024-03-20T08:47:23.957753683Z level=info msg="Executing migration" id="add index dashboard_permission" 2024-03-20T08:47:23.987 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Mar 20 08:47:23 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-grafana-a[51839]: logger=migrator t=2024-03-20T08:47:23.958175675Z level=info msg="Executing migration" id="save default acl rules in dashboard_acl table" 2024-03-20T08:47:23.987 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Mar 20 08:47:23 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-grafana-a[51839]: logger=migrator t=2024-03-20T08:47:23.958574624Z level=info msg="Executing migration" id="delete acl rules for deleted dashboards and folders" 2024-03-20T08:47:23.987 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Mar 20 08:47:23 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-grafana-a[51839]: logger=migrator t=2024-03-20T08:47:23.958845382Z level=info msg="Executing migration" id="create tag table" 2024-03-20T08:47:23.988 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Mar 20 08:47:23 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-grafana-a[51839]: logger=migrator t=2024-03-20T08:47:23.959228537Z level=info msg="Executing migration" id="add index tag.key_value" 2024-03-20T08:47:23.988 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Mar 20 08:47:23 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-grafana-a[51839]: logger=migrator t=2024-03-20T08:47:23.959641114Z level=info msg="Executing migration" id="create login attempt table" 2024-03-20T08:47:23.988 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Mar 20 08:47:23 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-grafana-a[51839]: logger=migrator t=2024-03-20T08:47:23.960016182Z level=info msg="Executing migration" id="add index login_attempt.username" 2024-03-20T08:47:23.988 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Mar 20 08:47:23 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-grafana-a[51839]: logger=migrator t=2024-03-20T08:47:23.960502399Z level=info msg="Executing migration" id="drop index IDX_login_attempt_username - v1" 2024-03-20T08:47:23.988 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Mar 20 08:47:23 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-grafana-a[51839]: logger=migrator t=2024-03-20T08:47:23.960894308Z level=info msg="Executing migration" id="Rename table login_attempt to login_attempt_tmp_qwerty - v1" 2024-03-20T08:47:23.988 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Mar 20 08:47:23 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-grafana-a[51839]: logger=migrator t=2024-03-20T08:47:23.962784893Z level=info msg="Executing migration" id="create login_attempt v2" 2024-03-20T08:47:23.988 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Mar 20 08:47:23 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-grafana-a[51839]: logger=migrator t=2024-03-20T08:47:23.963193632Z level=info msg="Executing migration" id="create index IDX_login_attempt_username - v2" 2024-03-20T08:47:23.988 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Mar 20 08:47:23 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-grafana-a[51839]: logger=migrator t=2024-03-20T08:47:23.963633307Z level=info msg="Executing migration" id="copy login_attempt v1 to v2" 2024-03-20T08:47:23.988 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Mar 20 08:47:23 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-grafana-a[51839]: logger=migrator t=2024-03-20T08:47:23.963982916Z level=info msg="Executing migration" id="drop login_attempt_tmp_qwerty" 2024-03-20T08:47:23.989 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Mar 20 08:47:23 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-grafana-a[51839]: logger=migrator t=2024-03-20T08:47:23.964376624Z level=info msg="Executing migration" id="create user auth table" 2024-03-20T08:47:23.989 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Mar 20 08:47:23 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-grafana-a[51839]: logger=migrator t=2024-03-20T08:47:23.964729673Z level=info msg="Executing migration" id="create index IDX_user_auth_auth_module_auth_id - v1" 2024-03-20T08:47:23.989 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Mar 20 08:47:23 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-grafana-a[51839]: logger=migrator t=2024-03-20T08:47:23.965164778Z level=info msg="Executing migration" id="alter user_auth.auth_id to length 190" 2024-03-20T08:47:23.989 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Mar 20 08:47:23 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-grafana-a[51839]: logger=migrator t=2024-03-20T08:47:23.965457628Z level=info msg="Executing migration" id="Add OAuth access token to user_auth" 2024-03-20T08:47:23.989 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Mar 20 08:47:23 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-grafana-a[51839]: logger=migrator t=2024-03-20T08:47:23.966184678Z level=info msg="Executing migration" id="Add OAuth refresh token to user_auth" 2024-03-20T08:47:23.989 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Mar 20 08:47:23 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-grafana-a[51839]: logger=migrator t=2024-03-20T08:47:23.966901121Z level=info msg="Executing migration" id="Add OAuth token type to user_auth" 2024-03-20T08:47:23.989 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Mar 20 08:47:23 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-grafana-a[51839]: logger=migrator t=2024-03-20T08:47:23.967842899Z level=info msg="Executing migration" id="Add OAuth expiry to user_auth" 2024-03-20T08:47:23.989 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Mar 20 08:47:23 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-grafana-a[51839]: logger=migrator t=2024-03-20T08:47:23.968661355Z level=info msg="Executing migration" id="Add index to user_id column in user_auth" 2024-03-20T08:47:23.989 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Mar 20 08:47:23 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-grafana-a[51839]: logger=migrator t=2024-03-20T08:47:23.969087337Z level=info msg="Executing migration" id="Add OAuth ID token to user_auth" 2024-03-20T08:47:23.989 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Mar 20 08:47:23 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-grafana-a[51839]: logger=migrator t=2024-03-20T08:47:23.969819352Z level=info msg="Executing migration" id="create server_lock table" 2024-03-20T08:47:23.989 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Mar 20 08:47:23 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-grafana-a[51839]: logger=migrator t=2024-03-20T08:47:23.970223134Z level=info msg="Executing migration" id="add index server_lock.operation_uid" 2024-03-20T08:47:23.989 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Mar 20 08:47:23 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-grafana-a[51839]: logger=migrator t=2024-03-20T08:47:23.970675027Z level=info msg="Executing migration" id="create user auth token table" 2024-03-20T08:47:23.989 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Mar 20 08:47:23 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-grafana-a[51839]: logger=migrator t=2024-03-20T08:47:23.971149155Z level=info msg="Executing migration" id="add unique index user_auth_token.auth_token" 2024-03-20T08:47:23.990 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Mar 20 08:47:23 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-grafana-a[51839]: logger=migrator t=2024-03-20T08:47:23.971586222Z level=info msg="Executing migration" id="add unique index user_auth_token.prev_auth_token" 2024-03-20T08:47:23.990 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Mar 20 08:47:23 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-grafana-a[51839]: logger=migrator t=2024-03-20T08:47:23.972011816Z level=info msg="Executing migration" id="add index user_auth_token.user_id" 2024-03-20T08:47:23.990 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Mar 20 08:47:23 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-grafana-a[51839]: logger=migrator t=2024-03-20T08:47:23.972444079Z level=info msg="Executing migration" id="Add revoked_at to the user auth token" 2024-03-20T08:47:23.990 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Mar 20 08:47:23 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-grafana-a[51839]: logger=migrator t=2024-03-20T08:47:23.973227156Z level=info msg="Executing migration" id="create cache_data table" 2024-03-20T08:47:23.990 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Mar 20 08:47:23 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-grafana-a[51839]: logger=migrator t=2024-03-20T08:47:23.973647876Z level=info msg="Executing migration" id="add unique index cache_data.cache_key" 2024-03-20T08:47:23.990 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Mar 20 08:47:23 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-grafana-a[51839]: logger=migrator t=2024-03-20T08:47:23.974094028Z level=info msg="Executing migration" id="create short_url table v1" 2024-03-20T08:47:23.990 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Mar 20 08:47:23 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-grafana-a[51839]: logger=migrator t=2024-03-20T08:47:23.974557543Z level=info msg="Executing migration" id="add index short_url.org_id-uid" 2024-03-20T08:47:23.990 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Mar 20 08:47:23 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-grafana-a[51839]: logger=migrator t=2024-03-20T08:47:23.977114058Z level=info msg="Executing migration" id="alter table short_url alter column created_by type to bigint" 2024-03-20T08:47:23.990 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Mar 20 08:47:23 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-grafana-a[51839]: logger=migrator t=2024-03-20T08:47:23.977474396Z level=info msg="Executing migration" id="delete alert_definition table" 2024-03-20T08:47:23.991 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Mar 20 08:47:23 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-grafana-a[51839]: logger=migrator t=2024-03-20T08:47:23.977777901Z level=info msg="Executing migration" id="recreate alert_definition table" 2024-03-20T08:47:23.991 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Mar 20 08:47:23 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-grafana-a[51839]: logger=migrator t=2024-03-20T08:47:23.978266009Z level=info msg="Executing migration" id="add index in alert_definition on org_id and title columns" 2024-03-20T08:47:23.991 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Mar 20 08:47:23 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-grafana-a[51839]: logger=migrator t=2024-03-20T08:47:23.980968797Z level=info msg="Executing migration" id="add index in alert_definition on org_id and uid columns" 2024-03-20T08:47:24.228 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Mar 20 08:47:23 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-grafana-a[51839]: logger=migrator t=2024-03-20T08:47:23.983766183Z level=info msg="Executing migration" id="alter alert_definition table data column to mediumtext in mysql" 2024-03-20T08:47:24.228 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Mar 20 08:47:23 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-grafana-a[51839]: logger=migrator t=2024-03-20T08:47:23.984110495Z level=info msg="Executing migration" id="drop index in alert_definition on org_id and title columns" 2024-03-20T08:47:24.228 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Mar 20 08:47:23 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-grafana-a[51839]: logger=migrator t=2024-03-20T08:47:23.986637877Z level=info msg="Executing migration" id="drop index in alert_definition on org_id and uid columns" 2024-03-20T08:47:24.228 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Mar 20 08:47:23 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-grafana-a[51839]: logger=migrator t=2024-03-20T08:47:23.987357972Z level=info msg="Executing migration" id="add unique index in alert_definition on org_id and title columns" 2024-03-20T08:47:24.228 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Mar 20 08:47:23 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-grafana-a[51839]: logger=migrator t=2024-03-20T08:47:23.988233061Z level=info msg="Executing migration" id="add unique index in alert_definition on org_id and uid columns" 2024-03-20T08:47:24.229 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Mar 20 08:47:23 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-grafana-a[51839]: logger=migrator t=2024-03-20T08:47:23.989073064Z level=info msg="Executing migration" id="Add column paused in alert_definition" 2024-03-20T08:47:24.229 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Mar 20 08:47:23 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-grafana-a[51839]: logger=migrator t=2024-03-20T08:47:23.989900472Z level=info msg="Executing migration" id="drop alert_definition table" 2024-03-20T08:47:24.229 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Mar 20 08:47:23 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-grafana-a[51839]: logger=migrator t=2024-03-20T08:47:23.990668353Z level=info msg="Executing migration" id="delete alert_definition_version table" 2024-03-20T08:47:24.229 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Mar 20 08:47:23 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-grafana-a[51839]: logger=migrator t=2024-03-20T08:47:23.99134565Z level=info msg="Executing migration" id="recreate alert_definition_version table" 2024-03-20T08:47:24.229 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Mar 20 08:47:23 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-grafana-a[51839]: logger=migrator t=2024-03-20T08:47:23.991822276Z level=info msg="Executing migration" id="add index in alert_definition_version table on alert_definition_id and version columns" 2024-03-20T08:47:24.229 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Mar 20 08:47:23 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-grafana-a[51839]: logger=migrator t=2024-03-20T08:47:23.99231907Z level=info msg="Executing migration" id="add index in alert_definition_version table on alert_definition_uid and version columns" 2024-03-20T08:47:24.229 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Mar 20 08:47:23 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-grafana-a[51839]: logger=migrator t=2024-03-20T08:47:23.992795348Z level=info msg="Executing migration" id="alter alert_definition_version table data column to mediumtext in mysql" 2024-03-20T08:47:24.229 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Mar 20 08:47:23 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-grafana-a[51839]: logger=migrator t=2024-03-20T08:47:23.993573745Z level=info msg="Executing migration" id="drop alert_definition_version table" 2024-03-20T08:47:24.229 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Mar 20 08:47:23 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-grafana-a[51839]: logger=migrator t=2024-03-20T08:47:23.994275599Z level=info msg="Executing migration" id="create alert_instance table" 2024-03-20T08:47:24.229 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Mar 20 08:47:23 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-grafana-a[51839]: logger=migrator t=2024-03-20T08:47:23.995126464Z level=info msg="Executing migration" id="add index in alert_instance table on def_org_id, def_uid and current_state columns" 2024-03-20T08:47:24.229 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Mar 20 08:47:23 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-grafana-a[51839]: logger=migrator t=2024-03-20T08:47:23.995719738Z level=info msg="Executing migration" id="add index in alert_instance table on def_org_id, current_state columns" 2024-03-20T08:47:24.229 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Mar 20 08:47:23 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-grafana-a[51839]: logger=migrator t=2024-03-20T08:47:23.996477599Z level=info msg="Executing migration" id="add column current_state_end to alert_instance" 2024-03-20T08:47:24.229 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Mar 20 08:47:23 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-grafana-a[51839]: logger=migrator t=2024-03-20T08:47:23.997338268Z level=info msg="Executing migration" id="remove index def_org_id, def_uid, current_state on alert_instance" 2024-03-20T08:47:24.229 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Mar 20 08:47:23 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-grafana-a[51839]: logger=migrator t=2024-03-20T08:47:23.998072681Z level=info msg="Executing migration" id="remove index def_org_id, current_state on alert_instance" 2024-03-20T08:47:24.230 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Mar 20 08:47:23 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-grafana-a[51839]: logger=migrator t=2024-03-20T08:47:23.998548756Z level=info msg="Executing migration" id="rename def_org_id to rule_org_id in alert_instance" 2024-03-20T08:47:24.230 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Mar 20 08:47:24 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-grafana-a[51839]: logger=migrator t=2024-03-20T08:47:24.001964236Z level=info msg="Executing migration" id="rename def_uid to rule_uid in alert_instance" 2024-03-20T08:47:24.230 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Mar 20 08:47:24 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-grafana-a[51839]: logger=migrator t=2024-03-20T08:47:24.005351075Z level=info msg="Executing migration" id="add index rule_org_id, rule_uid, current_state on alert_instance" 2024-03-20T08:47:24.230 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Mar 20 08:47:24 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-grafana-a[51839]: logger=migrator t=2024-03-20T08:47:24.006067083Z level=info msg="Executing migration" id="add index rule_org_id, current_state on alert_instance" 2024-03-20T08:47:24.230 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Mar 20 08:47:24 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-grafana-a[51839]: logger=migrator t=2024-03-20T08:47:24.006540972Z level=info msg="Executing migration" id="add current_reason column related to current_state" 2024-03-20T08:47:24.230 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Mar 20 08:47:24 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-grafana-a[51839]: logger=migrator t=2024-03-20T08:47:24.007431649Z level=info msg="Executing migration" id="create alert_rule table" 2024-03-20T08:47:24.230 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Mar 20 08:47:24 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-grafana-a[51839]: logger=migrator t=2024-03-20T08:47:24.008138145Z level=info msg="Executing migration" id="add index in alert_rule on org_id and title columns" 2024-03-20T08:47:24.230 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Mar 20 08:47:24 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-grafana-a[51839]: logger=migrator t=2024-03-20T08:47:24.008869949Z level=info msg="Executing migration" id="add index in alert_rule on org_id and uid columns" 2024-03-20T08:47:24.230 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Mar 20 08:47:24 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-grafana-a[51839]: logger=migrator t=2024-03-20T08:47:24.009355709Z level=info msg="Executing migration" id="add index in alert_rule on org_id, namespace_uid, group_uid columns" 2024-03-20T08:47:24.230 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Mar 20 08:47:24 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-grafana-a[51839]: logger=migrator t=2024-03-20T08:47:24.010067802Z level=info msg="Executing migration" id="alter alert_rule table data column to mediumtext in mysql" 2024-03-20T08:47:24.230 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Mar 20 08:47:24 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-grafana-a[51839]: logger=migrator t=2024-03-20T08:47:24.010779455Z level=info msg="Executing migration" id="add column for to alert_rule" 2024-03-20T08:47:24.231 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Mar 20 08:47:24 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-grafana-a[51839]: logger=migrator t=2024-03-20T08:47:24.01170148Z level=info msg="Executing migration" id="add column annotations to alert_rule" 2024-03-20T08:47:24.231 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Mar 20 08:47:24 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-grafana-a[51839]: logger=migrator t=2024-03-20T08:47:24.012599801Z level=info msg="Executing migration" id="add column labels to alert_rule" 2024-03-20T08:47:24.231 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Mar 20 08:47:24 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-grafana-a[51839]: logger=migrator t=2024-03-20T08:47:24.013496197Z level=info msg="Executing migration" id="remove unique index from alert_rule on org_id, title columns" 2024-03-20T08:47:24.231 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Mar 20 08:47:24 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-grafana-a[51839]: logger=migrator t=2024-03-20T08:47:24.014238479Z level=info msg="Executing migration" id="add index in alert_rule on org_id, namespase_uid and title columns" 2024-03-20T08:47:24.231 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Mar 20 08:47:24 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-grafana-a[51839]: logger=migrator t=2024-03-20T08:47:24.015007726Z level=info msg="Executing migration" id="add dashboard_uid column to alert_rule" 2024-03-20T08:47:24.232 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Mar 20 08:47:24 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-grafana-a[51839]: logger=migrator t=2024-03-20T08:47:24.015873335Z level=info msg="Executing migration" id="add panel_id column to alert_rule" 2024-03-20T08:47:24.232 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Mar 20 08:47:24 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-grafana-a[51839]: logger=migrator t=2024-03-20T08:47:24.018715679Z level=info msg="Executing migration" id="add index in alert_rule on org_id, dashboard_uid and panel_id columns" 2024-03-20T08:47:24.232 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Mar 20 08:47:24 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-grafana-a[51839]: logger=migrator t=2024-03-20T08:47:24.02133684Z level=info msg="Executing migration" id="add rule_group_idx column to alert_rule" 2024-03-20T08:47:24.233 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Mar 20 08:47:24 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-grafana-a[51839]: logger=migrator t=2024-03-20T08:47:24.024294625Z level=info msg="Executing migration" id="add is_paused column to alert_rule table" 2024-03-20T08:47:24.233 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Mar 20 08:47:24 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-grafana-a[51839]: logger=migrator t=2024-03-20T08:47:24.025345608Z level=info msg="Executing migration" id="fix is_paused column for alert_rule table" 2024-03-20T08:47:24.233 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Mar 20 08:47:24 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-grafana-a[51839]: logger=migrator t=2024-03-20T08:47:24.02806155Z level=info msg="Executing migration" id="create alert_rule_version table" 2024-03-20T08:47:24.233 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Mar 20 08:47:24 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-grafana-a[51839]: logger=migrator t=2024-03-20T08:47:24.03079478Z level=info msg="Executing migration" id="add index in alert_rule_version table on rule_org_id, rule_uid and version columns" 2024-03-20T08:47:24.233 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Mar 20 08:47:24 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-grafana-a[51839]: logger=migrator t=2024-03-20T08:47:24.039957714Z level=info msg="Executing migration" id="add index in alert_rule_version table on rule_org_id, rule_namespace_uid and rule_group columns" 2024-03-20T08:47:24.233 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Mar 20 08:47:24 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-grafana-a[51839]: logger=migrator t=2024-03-20T08:47:24.040493533Z level=info msg="Executing migration" id="alter alert_rule_version table data column to mediumtext in mysql" 2024-03-20T08:47:24.233 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Mar 20 08:47:24 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-grafana-a[51839]: logger=migrator t=2024-03-20T08:47:24.042325677Z level=info msg="Executing migration" id="add column for to alert_rule_version" 2024-03-20T08:47:24.233 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Mar 20 08:47:24 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-grafana-a[51839]: logger=migrator t=2024-03-20T08:47:24.043255379Z level=info msg="Executing migration" id="add column annotations to alert_rule_version" 2024-03-20T08:47:24.233 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Mar 20 08:47:24 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-grafana-a[51839]: logger=migrator t=2024-03-20T08:47:24.04421661Z level=info msg="Executing migration" id="add column labels to alert_rule_version" 2024-03-20T08:47:24.233 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Mar 20 08:47:24 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-grafana-a[51839]: logger=migrator t=2024-03-20T08:47:24.046115337Z level=info msg="Executing migration" id="add rule_group_idx column to alert_rule_version" 2024-03-20T08:47:24.233 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Mar 20 08:47:24 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-grafana-a[51839]: logger=migrator t=2024-03-20T08:47:24.047005004Z level=info msg="Executing migration" id="add is_paused column to alert_rule_versions table" 2024-03-20T08:47:24.234 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Mar 20 08:47:24 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-grafana-a[51839]: logger=migrator t=2024-03-20T08:47:24.048892707Z level=info msg="Executing migration" id="fix is_paused column for alert_rule_version table" 2024-03-20T08:47:24.234 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Mar 20 08:47:24 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-grafana-a[51839]: logger=migrator t=2024-03-20T08:47:24.050815263Z level=info msg="Executing migration" id=create_alert_configuration_table 2024-03-20T08:47:24.234 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Mar 20 08:47:24 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-grafana-a[51839]: logger=migrator t=2024-03-20T08:47:24.05267836Z level=info msg="Executing migration" id="Add column default in alert_configuration" 2024-03-20T08:47:24.234 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Mar 20 08:47:24 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-grafana-a[51839]: logger=migrator t=2024-03-20T08:47:24.053606922Z level=info msg="Executing migration" id="alert alert_configuration alertmanager_configuration column from TEXT to MEDIUMTEXT if mysql" 2024-03-20T08:47:24.234 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Mar 20 08:47:24 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-grafana-a[51839]: logger=migrator t=2024-03-20T08:47:24.05561263Z level=info msg="Executing migration" id="add column org_id in alert_configuration" 2024-03-20T08:47:24.234 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Mar 20 08:47:24 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-grafana-a[51839]: logger=migrator t=2024-03-20T08:47:24.057435994Z level=info msg="Executing migration" id="add index in alert_configuration table on org_id column" 2024-03-20T08:47:24.234 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Mar 20 08:47:24 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-grafana-a[51839]: logger=migrator t=2024-03-20T08:47:24.059335158Z level=info msg="Executing migration" id="add configuration_hash column to alert_configuration" 2024-03-20T08:47:24.234 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Mar 20 08:47:24 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-grafana-a[51839]: logger=migrator t=2024-03-20T08:47:24.060367532Z level=info msg="Executing migration" id=create_ngalert_configuration_table 2024-03-20T08:47:24.234 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Mar 20 08:47:24 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-grafana-a[51839]: logger=migrator t=2024-03-20T08:47:24.062280163Z level=info msg="Executing migration" id="add index in ngalert_configuration on org_id column" 2024-03-20T08:47:24.234 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Mar 20 08:47:24 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-grafana-a[51839]: logger=migrator t=2024-03-20T08:47:24.06421683Z level=info msg="Executing migration" id="add column send_alerts_to in ngalert_configuration" 2024-03-20T08:47:24.234 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Mar 20 08:47:24 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-grafana-a[51839]: logger=migrator t=2024-03-20T08:47:24.066208007Z level=info msg="Executing migration" id="create provenance_type table" 2024-03-20T08:47:24.234 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Mar 20 08:47:24 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-grafana-a[51839]: logger=migrator t=2024-03-20T08:47:24.066781005Z level=info msg="Executing migration" id="add index to uniquify (record_key, record_type, org_id) columns" 2024-03-20T08:47:24.235 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Mar 20 08:47:24 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-grafana-a[51839]: logger=migrator t=2024-03-20T08:47:24.068689639Z level=info msg="Executing migration" id="create alert_image table" 2024-03-20T08:47:24.235 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Mar 20 08:47:24 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-grafana-a[51839]: logger=migrator t=2024-03-20T08:47:24.070716203Z level=info msg="Executing migration" id="add unique index on token to alert_image table" 2024-03-20T08:47:24.235 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Mar 20 08:47:24 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-grafana-a[51839]: logger=migrator t=2024-03-20T08:47:24.072483984Z level=info msg="Executing migration" id="support longer URLs in alert_image table" 2024-03-20T08:47:24.235 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Mar 20 08:47:24 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-grafana-a[51839]: logger=migrator t=2024-03-20T08:47:24.072787434Z level=info msg="Executing migration" id=create_alert_configuration_history_table 2024-03-20T08:47:24.235 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Mar 20 08:47:24 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-grafana-a[51839]: logger=migrator t=2024-03-20T08:47:24.074796685Z level=info msg="Executing migration" id="drop non-unique orgID index on alert_configuration" 2024-03-20T08:47:24.235 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Mar 20 08:47:24 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-grafana-a[51839]: logger=migrator t=2024-03-20T08:47:24.076692633Z level=info msg="Executing migration" id="drop unique orgID index on alert_configuration if exists" 2024-03-20T08:47:24.235 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Mar 20 08:47:24 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-grafana-a[51839]: logger=migrator t=2024-03-20T08:47:24.076888032Z level=warn msg="Skipping migration: Already executed, but not recorded in migration log" id="drop unique orgID index on alert_configuration if exists" 2024-03-20T08:47:24.235 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Mar 20 08:47:24 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-grafana-a[51839]: logger=migrator t=2024-03-20T08:47:24.078559089Z level=info msg="Executing migration" id="extract alertmanager configuration history to separate table" 2024-03-20T08:47:24.235 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Mar 20 08:47:24 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-grafana-a[51839]: logger=migrator t=2024-03-20T08:47:24.078978627Z level=info msg="Executing migration" id="add unique index on orgID to alert_configuration" 2024-03-20T08:47:24.235 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Mar 20 08:47:24 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-grafana-a[51839]: logger=migrator t=2024-03-20T08:47:24.080876508Z level=info msg="Executing migration" id="clear migration entry \"remove unified alerting data\"" 2024-03-20T08:47:24.236 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Mar 20 08:47:24 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-grafana-a[51839]: logger=migrator t=2024-03-20T08:47:24.08101706Z level=info msg="Executing migration" id="move dashboard alerts to unified alerting" 2024-03-20T08:47:24.236 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Mar 20 08:47:24 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-grafana-a[51839]: logger=migrator t=2024-03-20T08:47:24.08112459Z level=info msg="alerts found to migrate" alerts=0 2024-03-20T08:47:24.236 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Mar 20 08:47:24 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-grafana-a[51839]: logger=migrator t=2024-03-20T08:47:24.082777655Z level=info msg="Executing migration" id="create library_element table v1" 2024-03-20T08:47:24.236 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Mar 20 08:47:24 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-grafana-a[51839]: logger=migrator t=2024-03-20T08:47:24.084791694Z level=info msg="Executing migration" id="add index library_element org_id-folder_id-name-kind" 2024-03-20T08:47:24.236 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Mar 20 08:47:24 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-grafana-a[51839]: logger=migrator t=2024-03-20T08:47:24.085391684Z level=info msg="Executing migration" id="create library_element_connection table v1" 2024-03-20T08:47:24.236 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Mar 20 08:47:24 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-grafana-a[51839]: logger=migrator t=2024-03-20T08:47:24.087311946Z level=info msg="Executing migration" id="add index library_element_connection element_id-kind-connection_id" 2024-03-20T08:47:24.236 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Mar 20 08:47:24 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-grafana-a[51839]: logger=migrator t=2024-03-20T08:47:24.089194378Z level=info msg="Executing migration" id="add unique index library_element org_id_uid" 2024-03-20T08:47:24.236 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Mar 20 08:47:24 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-grafana-a[51839]: logger=migrator t=2024-03-20T08:47:24.091467327Z level=info msg="Executing migration" id="increase max description length to 2048" 2024-03-20T08:47:24.236 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Mar 20 08:47:24 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-grafana-a[51839]: logger=migrator t=2024-03-20T08:47:24.091812862Z level=info msg="Executing migration" id="clone move dashboard alerts to unified alerting" 2024-03-20T08:47:24.236 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Mar 20 08:47:24 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-grafana-a[51839]: logger=migrator t=2024-03-20T08:47:24.093874668Z level=info msg="Executing migration" id="create data_keys table" 2024-03-20T08:47:24.236 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Mar 20 08:47:24 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-grafana-a[51839]: logger=migrator t=2024-03-20T08:47:24.09555942Z level=info msg="Executing migration" id="create secrets table" 2024-03-20T08:47:24.236 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Mar 20 08:47:24 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-grafana-a[51839]: logger=migrator t=2024-03-20T08:47:24.097370203Z level=info msg="Executing migration" id="rename data_keys name column to id" 2024-03-20T08:47:24.236 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Mar 20 08:47:24 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-grafana-a[51839]: logger=migrator t=2024-03-20T08:47:24.101691846Z level=info msg="Executing migration" id="add name column into data_keys" 2024-03-20T08:47:24.236 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Mar 20 08:47:24 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-grafana-a[51839]: logger=migrator t=2024-03-20T08:47:24.103640212Z level=info msg="Executing migration" id="copy data_keys id column values into name" 2024-03-20T08:47:24.237 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Mar 20 08:47:24 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-grafana-a[51839]: logger=migrator t=2024-03-20T08:47:24.105448746Z level=info msg="Executing migration" id="rename data_keys name column to label" 2024-03-20T08:47:24.237 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Mar 20 08:47:24 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-grafana-a[51839]: logger=migrator t=2024-03-20T08:47:24.109759304Z level=info msg="Executing migration" id="rename data_keys id column back to name" 2024-03-20T08:47:24.237 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Mar 20 08:47:24 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-grafana-a[51839]: logger=migrator t=2024-03-20T08:47:24.114027964Z level=info msg="Executing migration" id="create kv_store table v1" 2024-03-20T08:47:24.237 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Mar 20 08:47:24 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-grafana-a[51839]: logger=migrator t=2024-03-20T08:47:24.116033999Z level=info msg="Executing migration" id="add index kv_store.org_id-namespace-key" 2024-03-20T08:47:24.237 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Mar 20 08:47:24 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-grafana-a[51839]: logger=migrator t=2024-03-20T08:47:24.117865957Z level=info msg="Executing migration" id="update dashboard_uid and panel_id from existing annotations" 2024-03-20T08:47:24.237 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Mar 20 08:47:24 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-grafana-a[51839]: logger=migrator t=2024-03-20T08:47:24.119519015Z level=info msg="Executing migration" id="create permission table" 2024-03-20T08:47:24.237 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Mar 20 08:47:24 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-grafana-a[51839]: logger=migrator t=2024-03-20T08:47:24.119916286Z level=info msg="Executing migration" id="add unique index permission.role_id" 2024-03-20T08:47:24.237 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Mar 20 08:47:24 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-grafana-a[51839]: logger=migrator t=2024-03-20T08:47:24.121774688Z level=info msg="Executing migration" id="add unique index role_id_action_scope" 2024-03-20T08:47:24.237 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Mar 20 08:47:24 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-grafana-a[51839]: logger=migrator t=2024-03-20T08:47:24.123711535Z level=info msg="Executing migration" id="create role table" 2024-03-20T08:47:24.237 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Mar 20 08:47:24 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-grafana-a[51839]: logger=migrator t=2024-03-20T08:47:24.125526246Z level=info msg="Executing migration" id="add column display_name" 2024-03-20T08:47:24.237 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Mar 20 08:47:24 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-grafana-a[51839]: logger=migrator t=2024-03-20T08:47:24.126484868Z level=info msg="Executing migration" id="add column group_name" 2024-03-20T08:47:24.237 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Mar 20 08:47:24 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-grafana-a[51839]: logger=migrator t=2024-03-20T08:47:24.128322873Z level=info msg="Executing migration" id="add index role.org_id" 2024-03-20T08:47:24.237 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Mar 20 08:47:24 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-grafana-a[51839]: logger=migrator t=2024-03-20T08:47:24.130126227Z level=info msg="Executing migration" id="add unique index role_org_id_name" 2024-03-20T08:47:24.237 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Mar 20 08:47:24 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-grafana-a[51839]: logger=migrator t=2024-03-20T08:47:24.131986635Z level=info msg="Executing migration" id="add index role_org_id_uid" 2024-03-20T08:47:24.238 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Mar 20 08:47:24 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-grafana-a[51839]: logger=migrator t=2024-03-20T08:47:24.132413438Z level=info msg="Executing migration" id="create team role table" 2024-03-20T08:47:24.238 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Mar 20 08:47:24 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-grafana-a[51839]: logger=migrator t=2024-03-20T08:47:24.134305913Z level=info msg="Executing migration" id="add index team_role.org_id" 2024-03-20T08:47:24.238 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Mar 20 08:47:24 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-grafana-a[51839]: logger=migrator t=2024-03-20T08:47:24.136272783Z level=info msg="Executing migration" id="add unique index team_role_org_id_team_id_role_id" 2024-03-20T08:47:24.238 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Mar 20 08:47:24 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-grafana-a[51839]: logger=migrator t=2024-03-20T08:47:24.137986586Z level=info msg="Executing migration" id="add index team_role.team_id" 2024-03-20T08:47:24.238 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Mar 20 08:47:24 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-grafana-a[51839]: logger=migrator t=2024-03-20T08:47:24.139578164Z level=info msg="Executing migration" id="create user role table" 2024-03-20T08:47:24.238 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Mar 20 08:47:24 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-grafana-a[51839]: logger=migrator t=2024-03-20T08:47:24.140296Z level=info msg="Executing migration" id="add index user_role.org_id" 2024-03-20T08:47:24.238 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Mar 20 08:47:24 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-grafana-a[51839]: logger=migrator t=2024-03-20T08:47:24.142088951Z level=info msg="Executing migration" id="add unique index user_role_org_id_user_id_role_id" 2024-03-20T08:47:24.238 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Mar 20 08:47:24 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-grafana-a[51839]: logger=migrator t=2024-03-20T08:47:24.143882354Z level=info msg="Executing migration" id="add index user_role.user_id" 2024-03-20T08:47:24.238 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Mar 20 08:47:24 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-grafana-a[51839]: logger=migrator t=2024-03-20T08:47:24.144327817Z level=info msg="Executing migration" id="create builtin role table" 2024-03-20T08:47:24.238 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Mar 20 08:47:24 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-grafana-a[51839]: logger=migrator t=2024-03-20T08:47:24.146272153Z level=info msg="Executing migration" id="add index builtin_role.role_id" 2024-03-20T08:47:24.238 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Mar 20 08:47:24 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-grafana-a[51839]: logger=migrator t=2024-03-20T08:47:24.148144789Z level=info msg="Executing migration" id="add index builtin_role.name" 2024-03-20T08:47:24.238 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Mar 20 08:47:24 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-grafana-a[51839]: logger=migrator t=2024-03-20T08:47:24.149868976Z level=info msg="Executing migration" id="Add column org_id to builtin_role table" 2024-03-20T08:47:24.238 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Mar 20 08:47:24 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-grafana-a[51839]: logger=migrator t=2024-03-20T08:47:24.150884246Z level=info msg="Executing migration" id="add index builtin_role.org_id" 2024-03-20T08:47:24.238 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Mar 20 08:47:24 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-grafana-a[51839]: logger=migrator t=2024-03-20T08:47:24.152854794Z level=info msg="Executing migration" id="add unique index builtin_role_org_id_role_id_role" 2024-03-20T08:47:24.239 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Mar 20 08:47:24 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-grafana-a[51839]: logger=migrator t=2024-03-20T08:47:24.154660041Z level=info msg="Executing migration" id="Remove unique index role_org_id_uid" 2024-03-20T08:47:24.239 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Mar 20 08:47:24 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-grafana-a[51839]: logger=migrator t=2024-03-20T08:47:24.156452906Z level=info msg="Executing migration" id="add unique index role.uid" 2024-03-20T08:47:24.239 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Mar 20 08:47:24 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-grafana-a[51839]: logger=migrator t=2024-03-20T08:47:24.156941482Z level=info msg="Executing migration" id="create seed assignment table" 2024-03-20T08:47:24.239 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Mar 20 08:47:24 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-grafana-a[51839]: logger=migrator t=2024-03-20T08:47:24.158986957Z level=info msg="Executing migration" id="add unique index builtin_role_role_name" 2024-03-20T08:47:24.239 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Mar 20 08:47:24 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-grafana-a[51839]: logger=migrator t=2024-03-20T08:47:24.160872153Z level=info msg="Executing migration" id="add column hidden to role table" 2024-03-20T08:47:24.239 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Mar 20 08:47:24 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-grafana-a[51839]: logger=migrator t=2024-03-20T08:47:24.162620867Z level=info msg="Executing migration" id="create query_history table v1" 2024-03-20T08:47:24.239 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Mar 20 08:47:24 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-grafana-a[51839]: logger=migrator t=2024-03-20T08:47:24.163027811Z level=info msg="Executing migration" id="add index query_history.org_id-created_by-datasource_uid" 2024-03-20T08:47:24.239 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Mar 20 08:47:24 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-grafana-a[51839]: logger=migrator t=2024-03-20T08:47:24.165032416Z level=info msg="Executing migration" id="alter table query_history alter column created_by type to bigint" 2024-03-20T08:47:24.239 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Mar 20 08:47:24 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-grafana-a[51839]: logger=migrator t=2024-03-20T08:47:24.166824648Z level=info msg="Executing migration" id="teams permissions migration" 2024-03-20T08:47:24.239 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Mar 20 08:47:24 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-grafana-a[51839]: logger=migrator t=2024-03-20T08:47:24.168784438Z level=info msg="Executing migration" id="dashboard permissions" 2024-03-20T08:47:24.239 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Mar 20 08:47:24 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-grafana-a[51839]: logger=migrator t=2024-03-20T08:47:24.169408323Z level=info msg="Executing migration" id="dashboard permissions uid scopes" 2024-03-20T08:47:24.239 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Mar 20 08:47:24 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-grafana-a[51839]: logger=migrator t=2024-03-20T08:47:24.171261538Z level=info msg="Executing migration" id="drop managed folder create actions" 2024-03-20T08:47:24.239 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Mar 20 08:47:24 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-grafana-a[51839]: logger=migrator t=2024-03-20T08:47:24.171682511Z level=info msg="Executing migration" id="alerting notification permissions" 2024-03-20T08:47:24.239 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Mar 20 08:47:24 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-grafana-a[51839]: logger=migrator t=2024-03-20T08:47:24.173191662Z level=info msg="Executing migration" id="create query_history_star table v1" 2024-03-20T08:47:24.239 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Mar 20 08:47:24 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-grafana-a[51839]: logger=migrator t=2024-03-20T08:47:24.17506202Z level=info msg="Executing migration" id="add index query_history.user_id-query_uid" 2024-03-20T08:47:24.240 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Mar 20 08:47:24 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-grafana-a[51839]: logger=migrator t=2024-03-20T08:47:24.175493373Z level=info msg="Executing migration" id="add column org_id in query_history_star" 2024-03-20T08:47:24.240 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Mar 20 08:47:24 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-grafana-a[51839]: logger=migrator t=2024-03-20T08:47:24.177373858Z level=info msg="Executing migration" id="alter table query_history_star_mig column user_id type to bigint" 2024-03-20T08:47:24.240 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Mar 20 08:47:24 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-grafana-a[51839]: logger=migrator t=2024-03-20T08:47:24.179602628Z level=info msg="Executing migration" id="create correlation table v1" 2024-03-20T08:47:24.240 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Mar 20 08:47:24 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-grafana-a[51839]: logger=migrator t=2024-03-20T08:47:24.181546338Z level=info msg="Executing migration" id="add index correlations.uid" 2024-03-20T08:47:24.240 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Mar 20 08:47:24 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-grafana-a[51839]: logger=migrator t=2024-03-20T08:47:24.182107239Z level=info msg="Executing migration" id="add index correlations.source_uid" 2024-03-20T08:47:24.240 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Mar 20 08:47:24 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-grafana-a[51839]: logger=migrator t=2024-03-20T08:47:24.183994323Z level=info msg="Executing migration" id="add correlation config column" 2024-03-20T08:47:24.240 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Mar 20 08:47:24 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-grafana-a[51839]: logger=migrator t=2024-03-20T08:47:24.186047798Z level=info msg="Executing migration" id="create entity_events table" 2024-03-20T08:47:24.240 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Mar 20 08:47:24 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-grafana-a[51839]: logger=migrator t=2024-03-20T08:47:24.187792463Z level=info msg="Executing migration" id="create dashboard public config v1" 2024-03-20T08:47:24.240 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Mar 20 08:47:24 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-grafana-a[51839]: logger=migrator t=2024-03-20T08:47:24.18825224Z level=info msg="Executing migration" id="drop index UQE_dashboard_public_config_uid - v1" 2024-03-20T08:47:24.240 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Mar 20 08:47:24 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-grafana-a[51839]: logger=migrator t=2024-03-20T08:47:24.188336391Z level=warn msg="Skipping migration: Already executed, but not recorded in migration log" id="drop index UQE_dashboard_public_config_uid - v1" 2024-03-20T08:47:24.240 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Mar 20 08:47:24 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-grafana-a[51839]: logger=migrator t=2024-03-20T08:47:24.190135725Z level=info msg="Executing migration" id="drop index IDX_dashboard_public_config_org_id_dashboard_uid - v1" 2024-03-20T08:47:24.240 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Mar 20 08:47:24 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-grafana-a[51839]: logger=migrator t=2024-03-20T08:47:24.190212838Z level=warn msg="Skipping migration: Already executed, but not recorded in migration log" id="drop index IDX_dashboard_public_config_org_id_dashboard_uid - v1" 2024-03-20T08:47:24.240 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Mar 20 08:47:24 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-grafana-a[51839]: logger=migrator t=2024-03-20T08:47:24.192027561Z level=info msg="Executing migration" id="Drop old dashboard public config table" 2024-03-20T08:47:24.240 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Mar 20 08:47:24 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-grafana-a[51839]: logger=migrator t=2024-03-20T08:47:24.193904619Z level=info msg="Executing migration" id="recreate dashboard public config v1" 2024-03-20T08:47:24.240 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Mar 20 08:47:24 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-grafana-a[51839]: logger=migrator t=2024-03-20T08:47:24.194439282Z level=info msg="Executing migration" id="create index UQE_dashboard_public_config_uid - v1" 2024-03-20T08:47:24.241 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Mar 20 08:47:24 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-grafana-a[51839]: logger=migrator t=2024-03-20T08:47:24.195233187Z level=info msg="Executing migration" id="create index IDX_dashboard_public_config_org_id_dashboard_uid - v1" 2024-03-20T08:47:24.241 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Mar 20 08:47:24 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-grafana-a[51839]: logger=migrator t=2024-03-20T08:47:24.19610758Z level=info msg="Executing migration" id="drop index UQE_dashboard_public_config_uid - v2" 2024-03-20T08:47:24.241 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Mar 20 08:47:24 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-grafana-a[51839]: logger=migrator t=2024-03-20T08:47:24.197068387Z level=info msg="Executing migration" id="drop index IDX_dashboard_public_config_org_id_dashboard_uid - v2" 2024-03-20T08:47:24.241 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Mar 20 08:47:24 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-grafana-a[51839]: logger=migrator t=2024-03-20T08:47:24.197653291Z level=info msg="Executing migration" id="Drop public config table" 2024-03-20T08:47:24.241 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Mar 20 08:47:24 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-grafana-a[51839]: logger=migrator t=2024-03-20T08:47:24.198532569Z level=info msg="Executing migration" id="Recreate dashboard public config v2" 2024-03-20T08:47:24.241 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Mar 20 08:47:24 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-grafana-a[51839]: logger=migrator t=2024-03-20T08:47:24.199462222Z level=info msg="Executing migration" id="create index UQE_dashboard_public_config_uid - v2" 2024-03-20T08:47:24.241 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Mar 20 08:47:24 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-grafana-a[51839]: logger=migrator t=2024-03-20T08:47:24.200191779Z level=info msg="Executing migration" id="create index IDX_dashboard_public_config_org_id_dashboard_uid - v2" 2024-03-20T08:47:24.241 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Mar 20 08:47:24 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-grafana-a[51839]: logger=migrator t=2024-03-20T08:47:24.200648979Z level=info msg="Executing migration" id="create index UQE_dashboard_public_config_access_token - v2" 2024-03-20T08:47:24.241 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Mar 20 08:47:24 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-grafana-a[51839]: logger=migrator t=2024-03-20T08:47:24.201617785Z level=info msg="Executing migration" id="Rename table dashboard_public_config to dashboard_public - v2" 2024-03-20T08:47:24.241 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Mar 20 08:47:24 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-grafana-a[51839]: logger=migrator t=2024-03-20T08:47:24.204705989Z level=info msg="Executing migration" id="add annotations_enabled column" 2024-03-20T08:47:24.241 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Mar 20 08:47:24 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-grafana-a[51839]: logger=migrator t=2024-03-20T08:47:24.205704597Z level=info msg="Executing migration" id="add time_selection_enabled column" 2024-03-20T08:47:24.241 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Mar 20 08:47:24 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-grafana-a[51839]: logger=migrator t=2024-03-20T08:47:24.206717174Z level=info msg="Executing migration" id="delete orphaned public dashboards" 2024-03-20T08:47:24.241 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Mar 20 08:47:24 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-grafana-a[51839]: logger=migrator t=2024-03-20T08:47:24.207657822Z level=info msg="Executing migration" id="add share column" 2024-03-20T08:47:24.241 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Mar 20 08:47:24 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-grafana-a[51839]: logger=migrator t=2024-03-20T08:47:24.208709836Z level=info msg="Executing migration" id="create default alerting folders" 2024-03-20T08:47:24.241 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Mar 20 08:47:24 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-grafana-a[51839]: logger=migrator t=2024-03-20T08:47:24.209651817Z level=info msg="Executing migration" id="create file table" 2024-03-20T08:47:24.242 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Mar 20 08:47:24 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-grafana-a[51839]: logger=migrator t=2024-03-20T08:47:24.210203909Z level=info msg="Executing migration" id="file table idx: path natural pk" 2024-03-20T08:47:24.242 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Mar 20 08:47:24 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-grafana-a[51839]: logger=migrator t=2024-03-20T08:47:24.211176994Z level=info msg="Executing migration" id="file table idx: parent_folder_path_hash fast folder retrieval" 2024-03-20T08:47:24.242 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Mar 20 08:47:24 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-grafana-a[51839]: logger=migrator t=2024-03-20T08:47:24.211887844Z level=info msg="Executing migration" id="create file_meta table" 2024-03-20T08:47:24.242 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Mar 20 08:47:24 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-grafana-a[51839]: logger=migrator t=2024-03-20T08:47:24.212321956Z level=info msg="Executing migration" id="file table idx: path key" 2024-03-20T08:47:24.242 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Mar 20 08:47:24 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-grafana-a[51839]: logger=migrator t=2024-03-20T08:47:24.212806494Z level=info msg="Executing migration" id="set path collation in file table" 2024-03-20T08:47:24.242 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Mar 20 08:47:24 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-grafana-a[51839]: logger=migrator t=2024-03-20T08:47:24.214260092Z level=info msg="Executing migration" id="managed permissions migration" 2024-03-20T08:47:24.242 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Mar 20 08:47:24 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-grafana-a[51839]: logger=migrator t=2024-03-20T08:47:24.216439946Z level=info msg="Executing migration" id="managed folder permissions alert actions migration" 2024-03-20T08:47:24.242 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Mar 20 08:47:24 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-grafana-a[51839]: logger=migrator t=2024-03-20T08:47:24.219287115Z level=info msg="Executing migration" id="RBAC action name migrator" 2024-03-20T08:47:24.242 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Mar 20 08:47:24 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-grafana-a[51839]: logger=migrator t=2024-03-20T08:47:24.219724442Z level=info msg="Executing migration" id="Add UID column to playlist" 2024-03-20T08:47:24.242 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Mar 20 08:47:24 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-grafana-a[51839]: logger=migrator t=2024-03-20T08:47:24.222462443Z level=info msg="Executing migration" id="Update uid column values in playlist" 2024-03-20T08:47:24.242 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Mar 20 08:47:24 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-grafana-a[51839]: logger=migrator t=2024-03-20T08:47:24.225452213Z level=info msg="Executing migration" id="Add index for uid in playlist" 2024-03-20T08:47:24.242 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Mar 20 08:47:24 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-grafana-a[51839]: logger=migrator t=2024-03-20T08:47:24.228120406Z level=info msg="Executing migration" id="update group index for alert rules" 2024-03-20T08:47:24.242 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Mar 20 08:47:24 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-grafana-a[51839]: logger=migrator t=2024-03-20T08:47:24.228563334Z level=info msg="Executing migration" id="managed folder permissions alert actions repeated migration" 2024-03-20T08:47:24.242 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Mar 20 08:47:24 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-grafana-a[51839]: logger=migrator t=2024-03-20T08:47:24.231255754Z level=info msg="Executing migration" id="admin only folder/dashboard permission" 2024-03-20T08:47:24.242 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Mar 20 08:47:24 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-grafana-a[51839]: logger=migrator t=2024-03-20T08:47:24.231667916Z level=info msg="Executing migration" id="add action column to seed_assignment" 2024-03-20T08:47:24.531 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Mar 20 08:47:24 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-grafana-a[51839]: logger=migrator t=2024-03-20T08:47:24.234093346Z level=info msg="Executing migration" id="add scope column to seed_assignment" 2024-03-20T08:47:24.532 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Mar 20 08:47:24 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-grafana-a[51839]: logger=migrator t=2024-03-20T08:47:24.236896Z level=info msg="Executing migration" id="remove unique index builtin_role_role_name before nullable update" 2024-03-20T08:47:24.532 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Mar 20 08:47:24 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-grafana-a[51839]: logger=migrator t=2024-03-20T08:47:24.237413357Z level=info msg="Executing migration" id="update seed_assignment role_name column to nullable" 2024-03-20T08:47:24.532 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Mar 20 08:47:24 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-grafana-a[51839]: logger=migrator t=2024-03-20T08:47:24.248124203Z level=info msg="Executing migration" id="add unique index builtin_role_name back" 2024-03-20T08:47:24.532 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Mar 20 08:47:24 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-grafana-a[51839]: logger=migrator t=2024-03-20T08:47:24.250993426Z level=info msg="Executing migration" id="add unique index builtin_role_action_scope" 2024-03-20T08:47:24.532 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Mar 20 08:47:24 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-grafana-a[51839]: logger=migrator t=2024-03-20T08:47:24.253893449Z level=info msg="Executing migration" id="add primary key to seed_assigment" 2024-03-20T08:47:24.532 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Mar 20 08:47:24 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-grafana-a[51839]: logger=migrator t=2024-03-20T08:47:24.257333833Z level=info msg="Executing migration" id="managed folder permissions alert actions repeated fixed migration" 2024-03-20T08:47:24.532 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Mar 20 08:47:24 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-grafana-a[51839]: logger=migrator t=2024-03-20T08:47:24.260134017Z level=info msg="Executing migration" id="migrate external alertmanagers to datsourcse" 2024-03-20T08:47:24.532 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Mar 20 08:47:24 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-grafana-a[51839]: logger=migrator t=2024-03-20T08:47:24.263102607Z level=info msg="Executing migration" id="create folder table" 2024-03-20T08:47:24.532 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Mar 20 08:47:24 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-grafana-a[51839]: logger=migrator t=2024-03-20T08:47:24.26586812Z level=info msg="Executing migration" id="Add index for parent_uid" 2024-03-20T08:47:24.532 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Mar 20 08:47:24 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-grafana-a[51839]: logger=migrator t=2024-03-20T08:47:24.266346587Z level=info msg="Executing migration" id="Add unique index for folder.uid and folder.org_id" 2024-03-20T08:47:24.532 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Mar 20 08:47:24 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-grafana-a[51839]: logger=migrator t=2024-03-20T08:47:24.269109432Z level=info msg="Executing migration" id="Update folder title length" 2024-03-20T08:47:24.532 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Mar 20 08:47:24 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-grafana-a[51839]: logger=migrator t=2024-03-20T08:47:24.271687264Z level=info msg="Executing migration" id="Add unique index for folder.title and folder.parent_uid" 2024-03-20T08:47:24.533 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Mar 20 08:47:24 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-grafana-a[51839]: logger=migrator t=2024-03-20T08:47:24.274396084Z level=info msg="migrations completed" performed=485 skipped=0 duration=745.821408ms 2024-03-20T08:47:24.533 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Mar 20 08:47:24 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-grafana-a[51839]: logger=sqlstore t=2024-03-20T08:47:24.274803737Z level=info msg="Created default organization" 2024-03-20T08:47:24.533 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Mar 20 08:47:24 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-grafana-a[51839]: logger=plugin.loader t=2024-03-20T08:47:24.333479018Z level=info msg="Plugin registered" pluginID=input 2024-03-20T08:47:24.533 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Mar 20 08:47:24 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-grafana-a[51839]: logger=plugin.loader t=2024-03-20T08:47:24.362193187Z level=info msg="Plugin registered" pluginID=grafana-piechart-panel 2024-03-20T08:47:24.533 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Mar 20 08:47:24 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-grafana-a[51839]: logger=plugin.loader t=2024-03-20T08:47:24.362436659Z level=info msg="Plugin registered" pluginID=vonage-status-panel 2024-03-20T08:47:24.533 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Mar 20 08:47:24 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-grafana-a[51839]: logger=secrets t=2024-03-20T08:47:24.362608816Z level=info msg="Envelope encryption state" enabled=true currentprovider=secretKey.v1 2024-03-20T08:47:24.533 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Mar 20 08:47:24 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-grafana-a[51839]: logger=query_data t=2024-03-20T08:47:24.365573805Z level=info msg="Query Service initialization" 2024-03-20T08:47:24.533 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Mar 20 08:47:24 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-grafana-a[51839]: logger=live.push_http t=2024-03-20T08:47:24.368433199Z level=info msg="Live Push Gateway initialization" 2024-03-20T08:47:24.839 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:47:24 smithi194 ceph-mon[27237]: from='mgr.14637 ' entity='mgr.y' 2024-03-20T08:47:24.839 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:47:24 smithi194 ceph-mon[27237]: from='mgr.14637 ' entity='mgr.y' 2024-03-20T08:47:24.840 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:47:24 smithi194 ceph-mon[27237]: from='mgr.14637 ' entity='mgr.y' 2024-03-20T08:47:24.840 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:47:24 smithi194 ceph-mon[27237]: from='mgr.14637 ' entity='mgr.y' 2024-03-20T08:47:24.840 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:47:24 smithi194 ceph-mon[27237]: Deploying daemon node-exporter.a on smithi183 2024-03-20T08:47:24.840 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:47:24 smithi194 ceph-mon[27237]: pgmap v30: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-20T08:47:24.933 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:47:24 smithi183 ceph-mon[31860]: from='mgr.14637 ' entity='mgr.y' 2024-03-20T08:47:24.933 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:47:24 smithi183 ceph-mon[31860]: from='mgr.14637 ' entity='mgr.y' 2024-03-20T08:47:24.933 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:47:24 smithi183 ceph-mon[31860]: from='mgr.14637 ' entity='mgr.y' 2024-03-20T08:47:24.934 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:47:24 smithi183 ceph-mon[31860]: from='mgr.14637 ' entity='mgr.y' 2024-03-20T08:47:24.934 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:47:24 smithi183 ceph-mon[31860]: Deploying daemon node-exporter.a on smithi183 2024-03-20T08:47:24.934 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:47:24 smithi183 ceph-mon[31860]: pgmap v30: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-20T08:47:24.934 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:47:24 smithi183 ceph-mon[25285]: from='mgr.14637 ' entity='mgr.y' 2024-03-20T08:47:24.934 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:47:24 smithi183 ceph-mon[25285]: from='mgr.14637 ' entity='mgr.y' 2024-03-20T08:47:24.934 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:47:24 smithi183 ceph-mon[25285]: from='mgr.14637 ' entity='mgr.y' 2024-03-20T08:47:24.934 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:47:24 smithi183 ceph-mon[25285]: from='mgr.14637 ' entity='mgr.y' 2024-03-20T08:47:24.934 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:47:24 smithi183 ceph-mon[25285]: Deploying daemon node-exporter.a on smithi183 2024-03-20T08:47:24.934 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:47:24 smithi183 ceph-mon[25285]: pgmap v30: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-20T08:47:25.036 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-20T08:47:25.036 INFO:teuthology.orchestra.run.smithi183.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-03-20T08:47:25.093 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Mar 20 08:47:25 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-grafana-a[51839]: logger=infra.usagestats.collector t=2024-03-20T08:47:25.015384322Z level=info msg="registering usage stat providers" usageStatsProvidersLen=2 2024-03-20T08:47:25.093 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Mar 20 08:47:25 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-grafana-a[51839]: logger=provisioning.datasources t=2024-03-20T08:47:25.016413656Z level=info msg="inserting datasource from configuration " name=Dashboard1 uid=P43CA22E17D0F9596 2024-03-20T08:47:25.093 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Mar 20 08:47:25 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-grafana-a[51839]: logger=provisioning.datasources t=2024-03-20T08:47:25.027623324Z level=info msg="inserting datasource from configuration " name=Loki uid=P8E80F9AEF21F6940 2024-03-20T08:47:25.093 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Mar 20 08:47:25 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-grafana-a[51839]: logger=provisioning.alerting t=2024-03-20T08:47:25.038758057Z level=info msg="starting to provision alerting" 2024-03-20T08:47:25.094 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Mar 20 08:47:25 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-grafana-a[51839]: logger=provisioning.alerting t=2024-03-20T08:47:25.038780557Z level=info msg="finished to provision alerting" 2024-03-20T08:47:25.094 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Mar 20 08:47:25 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-grafana-a[51839]: logger=ngalert.state.manager t=2024-03-20T08:47:25.038955418Z level=info msg="Warming state cache for startup" 2024-03-20T08:47:25.094 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Mar 20 08:47:25 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-grafana-a[51839]: logger=grafanaStorageLogger t=2024-03-20T08:47:25.039166928Z level=info msg="storage starting" 2024-03-20T08:47:25.094 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Mar 20 08:47:25 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-grafana-a[51839]: logger=http.server t=2024-03-20T08:47:25.041066641Z level=info msg="HTTP Server Listen" address=[::]:3000 protocol=https subUrl= socket= 2024-03-20T08:47:25.094 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Mar 20 08:47:25 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-grafana-a[51839]: logger=ngalert.state.manager t=2024-03-20T08:47:25.044637945Z level=info msg="State cache has been initialized" states=0 duration=5.681805ms 2024-03-20T08:47:25.094 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Mar 20 08:47:25 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-grafana-a[51839]: logger=ticker t=2024-03-20T08:47:25.044705358Z level=info msg=starting first_tick=2024-03-20T08:47:30Z 2024-03-20T08:47:25.094 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Mar 20 08:47:25 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-grafana-a[51839]: logger=ngalert.multiorg.alertmanager t=2024-03-20T08:47:25.044726106Z level=info msg="starting MultiOrg Alertmanager" 2024-03-20T08:47:25.094 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Mar 20 08:47:25 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-grafana-a[51839]: logger=sqlstore.transactions t=2024-03-20T08:47:25.049995215Z level=info msg="Database locked, sleeping then retrying" error="database is locked" retry=0 code="database is locked" 2024-03-20T08:47:25.094 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Mar 20 08:47:25 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-grafana-a[51839]: logger=sqlstore.transactions t=2024-03-20T08:47:25.072773157Z level=info msg="Database locked, sleeping then retrying" error="database is locked" retry=1 code="database is locked" 2024-03-20T08:47:25.094 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Mar 20 08:47:25 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-grafana-a[51839]: logger=sqlstore.transactions t=2024-03-20T08:47:25.072782497Z level=info msg="Database locked, sleeping then retrying" error="database is locked" retry=0 code="database is locked" 2024-03-20T08:47:25.094 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Mar 20 08:47:25 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-grafana-a[51839]: logger=sqlstore.transactions t=2024-03-20T08:47:25.083022298Z level=info msg="Database locked, sleeping then retrying" error="database is locked" retry=2 code="database is locked" 2024-03-20T08:47:25.094 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Mar 20 08:47:25 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-grafana-a[51839]: logger=sqlstore.transactions t=2024-03-20T08:47:25.093203867Z level=info msg="Database locked, sleeping then retrying" error="database is locked" retry=3 code="database is locked" 2024-03-20T08:47:25.826 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:47:25 smithi183 ceph-mon[25285]: from='mgr.14637 ' entity='mgr.y' 2024-03-20T08:47:25.826 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:47:25 smithi183 ceph-mon[25285]: from='client.? 172.21.15.183:0/4015432915' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-03-20T08:47:25.826 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:47:25 smithi183 ceph-mon[31860]: from='mgr.14637 ' entity='mgr.y' 2024-03-20T08:47:25.826 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:47:25 smithi183 ceph-mon[31860]: from='client.? 172.21.15.183:0/4015432915' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-03-20T08:47:25.827 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy done 2024-03-20T08:47:25.827 INFO:tasks.cephadm:Setup complete, yielding 2024-03-20T08:47:25.828 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-03-20T08:47:25.837 INFO:tasks.cephadm:Running commands on role mon.a host ubuntu@smithi183.front.sepia.ceph.com 2024-03-20T08:47:25.838 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4c3f3eba-e695-11ee-95c9-87774f69a715 -- bash -c 'stat -c '"'"'%u %g'"'"' /var/log/ceph | grep '"'"'167 167'"'"'' 2024-03-20T08:47:26.122 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/4c3f3eba-e695-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T08:47:26.139 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:47:25 smithi194 ceph-mon[27237]: from='mgr.14637 ' entity='mgr.y' 2024-03-20T08:47:26.139 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:47:25 smithi194 ceph-mon[27237]: from='client.? 172.21.15.183:0/4015432915' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-03-20T08:47:26.204 INFO:journalctl@ceph.node-exporter.a.smithi183.stdout:Mar 20 08:47:26 smithi183 bash[61994]: Trying to pull quay.io/prometheus/node-exporter:v1.5.0... 2024-03-20T08:47:26.790 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:47:26 smithi183 ceph-mon[25285]: pgmap v31: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-20T08:47:26.790 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:47:26 smithi183 ceph-mon[31860]: pgmap v31: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-20T08:47:26.837 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:47:26 smithi194 ceph-mon[27237]: pgmap v31: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-20T08:47:27.204 INFO:journalctl@ceph.mgr.y.smithi183.stdout:Mar 20 08:47:26 smithi183 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-mgr-y[25513]: ::ffff:172.21.15.194 - - [20/Mar/2024:08:47:26] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-20T08:47:27.205 INFO:journalctl@ceph.node-exporter.a.smithi183.stdout:Mar 20 08:47:26 smithi183 bash[61994]: Getting image source signatures 2024-03-20T08:47:27.205 INFO:journalctl@ceph.node-exporter.a.smithi183.stdout:Mar 20 08:47:26 smithi183 bash[61994]: Copying blob sha256:c0e87333d3805a7b7e1c25783dd25f27da0fe70a8e262eee7ed190c8dc843fa7 2024-03-20T08:47:27.205 INFO:journalctl@ceph.node-exporter.a.smithi183.stdout:Mar 20 08:47:26 smithi183 bash[61994]: Copying blob sha256:22b70bddd3acadc892fca4c2af4260629bfda5dfd11ebc106a93ce24e752b5ed 2024-03-20T08:47:27.205 INFO:journalctl@ceph.node-exporter.a.smithi183.stdout:Mar 20 08:47:26 smithi183 bash[61994]: Copying blob sha256:5c12815fee558b157dee7f7509dedbaba0a8379098858a65ec869e1f1526ea0c 2024-03-20T08:47:28.979 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:47:28 smithi183 ceph-mon[25285]: pgmap v32: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-20T08:47:28.979 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:47:28 smithi183 ceph-mon[31860]: pgmap v32: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-20T08:47:28.982 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:47:28 smithi194 ceph-mon[27237]: pgmap v32: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-20T08:47:29.070 INFO:teuthology.orchestra.run.smithi183.stdout:167 167 2024-03-20T08:47:30.010 INFO:journalctl@ceph.node-exporter.a.smithi183.stdout:Mar 20 08:47:29 smithi183 bash[61994]: Copying config sha256:0da6a335fe1356545476b749c68f022c897de3a2139e8f0054f6937349ee2b83 2024-03-20T08:47:30.011 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:47:29 smithi183 ceph-mon[25285]: from='mgr.14637 172.21.15.183:0/885080955' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.2", "id": [1, 2]}]: dispatch 2024-03-20T08:47:30.011 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:47:29 smithi183 ceph-mon[25285]: from='mgr.14637 172.21.15.183:0/885080955' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.b", "id": [1, 2]}]: dispatch 2024-03-20T08:47:30.011 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:47:29 smithi183 ceph-mon[25285]: from='mgr.14637 172.21.15.183:0/885080955' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1b", "id": [1, 5]}]: dispatch 2024-03-20T08:47:30.011 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:47:29 smithi183 ceph-mon[25285]: from='mgr.14637 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.2", "id": [1, 2]}]: dispatch 2024-03-20T08:47:30.011 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:47:29 smithi183 ceph-mon[25285]: from='mgr.14637 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.b", "id": [1, 2]}]: dispatch 2024-03-20T08:47:30.011 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:47:29 smithi183 ceph-mon[25285]: from='mgr.14637 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1b", "id": [1, 5]}]: dispatch 2024-03-20T08:47:30.011 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:47:29 smithi183 ceph-mon[31860]: from='mgr.14637 172.21.15.183:0/885080955' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.2", "id": [1, 2]}]: dispatch 2024-03-20T08:47:30.011 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:47:29 smithi183 ceph-mon[31860]: from='mgr.14637 172.21.15.183:0/885080955' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.b", "id": [1, 2]}]: dispatch 2024-03-20T08:47:30.011 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:47:29 smithi183 ceph-mon[31860]: from='mgr.14637 172.21.15.183:0/885080955' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1b", "id": [1, 5]}]: dispatch 2024-03-20T08:47:30.011 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:47:29 smithi183 ceph-mon[31860]: from='mgr.14637 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.2", "id": [1, 2]}]: dispatch 2024-03-20T08:47:30.011 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:47:29 smithi183 ceph-mon[31860]: from='mgr.14637 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.b", "id": [1, 2]}]: dispatch 2024-03-20T08:47:30.012 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:47:29 smithi183 ceph-mon[31860]: from='mgr.14637 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1b", "id": [1, 5]}]: dispatch 2024-03-20T08:47:30.014 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:47:29 smithi194 ceph-mon[27237]: from='mgr.14637 172.21.15.183:0/885080955' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.2", "id": [1, 2]}]: dispatch 2024-03-20T08:47:30.014 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:47:29 smithi194 ceph-mon[27237]: from='mgr.14637 172.21.15.183:0/885080955' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.b", "id": [1, 2]}]: dispatch 2024-03-20T08:47:30.014 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:47:29 smithi194 ceph-mon[27237]: from='mgr.14637 172.21.15.183:0/885080955' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1b", "id": [1, 5]}]: dispatch 2024-03-20T08:47:30.014 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:47:29 smithi194 ceph-mon[27237]: from='mgr.14637 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.2", "id": [1, 2]}]: dispatch 2024-03-20T08:47:30.014 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:47:29 smithi194 ceph-mon[27237]: from='mgr.14637 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.b", "id": [1, 2]}]: dispatch 2024-03-20T08:47:30.014 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:47:29 smithi194 ceph-mon[27237]: from='mgr.14637 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1b", "id": [1, 5]}]: dispatch 2024-03-20T08:47:30.381 INFO:journalctl@ceph.node-exporter.a.smithi183.stdout:Mar 20 08:47:30 smithi183 bash[61994]: Writing manifest to image destination 2024-03-20T08:47:30.648 INFO:journalctl@ceph.node-exporter.a.smithi183.stdout:Mar 20 08:47:30 smithi183 podman[61994]: 2024-03-20 08:47:30.325559551 +0000 UTC m=+4.375342881 image pull 0da6a335fe1356545476b749c68f022c897de3a2139e8f0054f6937349ee2b83 quay.io/prometheus/node-exporter:v1.5.0 2024-03-20T08:47:30.648 INFO:journalctl@ceph.node-exporter.a.smithi183.stdout:Mar 20 08:47:30 smithi183 podman[61994]: 2024-03-20 08:47:30.558029856 +0000 UTC m=+4.607813229 container create 0dace97e4fc4562217de3d754934c883b3949f55529b51ef7f64abc4d49e3d5b (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-node-exporter-a, maintainer=The Prometheus Authors ) 2024-03-20T08:47:30.995 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:47:30 smithi194 ceph-mon[27237]: pgmap v33: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-20T08:47:30.995 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:47:30 smithi194 ceph-mon[27237]: from='mgr.14637 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.2", "id": [1, 2]}]': finished 2024-03-20T08:47:30.995 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:47:30 smithi194 ceph-mon[27237]: from='mgr.14637 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.b", "id": [1, 2]}]': finished 2024-03-20T08:47:30.995 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:47:30 smithi194 ceph-mon[27237]: from='mgr.14637 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1b", "id": [1, 5]}]': finished 2024-03-20T08:47:30.996 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:47:30 smithi194 ceph-mon[27237]: osdmap e70: 8 total, 8 up, 8 in 2024-03-20T08:47:30.996 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:47:30 smithi194 ceph-mon[27237]: from='mgr.14637 172.21.15.183:0/885080955' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-20T08:47:31.070 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:47:30 smithi183 ceph-mon[25285]: pgmap v33: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-20T08:47:31.070 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:47:30 smithi183 ceph-mon[25285]: from='mgr.14637 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.2", "id": [1, 2]}]': finished 2024-03-20T08:47:31.071 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:47:30 smithi183 ceph-mon[25285]: from='mgr.14637 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.b", "id": [1, 2]}]': finished 2024-03-20T08:47:31.071 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:47:30 smithi183 ceph-mon[25285]: from='mgr.14637 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1b", "id": [1, 5]}]': finished 2024-03-20T08:47:31.071 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:47:30 smithi183 ceph-mon[25285]: osdmap e70: 8 total, 8 up, 8 in 2024-03-20T08:47:31.071 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:47:30 smithi183 ceph-mon[25285]: from='mgr.14637 172.21.15.183:0/885080955' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-20T08:47:31.071 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:47:30 smithi183 ceph-mon[31860]: pgmap v33: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-20T08:47:31.071 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:47:30 smithi183 ceph-mon[31860]: from='mgr.14637 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.2", "id": [1, 2]}]': finished 2024-03-20T08:47:31.071 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:47:30 smithi183 ceph-mon[31860]: from='mgr.14637 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.b", "id": [1, 2]}]': finished 2024-03-20T08:47:31.071 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:47:30 smithi183 ceph-mon[31860]: from='mgr.14637 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1b", "id": [1, 5]}]': finished 2024-03-20T08:47:31.071 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:47:30 smithi183 ceph-mon[31860]: osdmap e70: 8 total, 8 up, 8 in 2024-03-20T08:47:31.071 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:47:30 smithi183 ceph-mon[31860]: from='mgr.14637 172.21.15.183:0/885080955' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-20T08:47:31.103 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4c3f3eba-e695-11ee-95c9-87774f69a715 -- bash -c 'ceph orch status' 2024-03-20T08:47:31.342 INFO:journalctl@ceph.node-exporter.a.smithi183.stdout:Mar 20 08:47:31 smithi183 podman[61994]: 2024-03-20 08:47:31.188703522 +0000 UTC m=+5.238486852 container init 0dace97e4fc4562217de3d754934c883b3949f55529b51ef7f64abc4d49e3d5b (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-node-exporter-a, maintainer=The Prometheus Authors ) 2024-03-20T08:47:31.342 INFO:journalctl@ceph.node-exporter.a.smithi183.stdout:Mar 20 08:47:31 smithi183 podman[61994]: 2024-03-20 08:47:31.191537061 +0000 UTC m=+5.241320396 container start 0dace97e4fc4562217de3d754934c883b3949f55529b51ef7f64abc4d49e3d5b (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-node-exporter-a, maintainer=The Prometheus Authors ) 2024-03-20T08:47:31.342 INFO:journalctl@ceph.node-exporter.a.smithi183.stdout:Mar 20 08:47:31 smithi183 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-node-exporter-a[62213]: ts=2024-03-20T08:47:31.195Z caller=node_exporter.go:180 level=info msg="Starting node_exporter" version="(version=1.5.0, branch=HEAD, revision=1b48970ffcf5630534fb00bb0687d73c66d1c959)" 2024-03-20T08:47:31.342 INFO:journalctl@ceph.node-exporter.a.smithi183.stdout:Mar 20 08:47:31 smithi183 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-node-exporter-a[62213]: ts=2024-03-20T08:47:31.195Z caller=node_exporter.go:181 level=info msg="Build context" build_context="(go=go1.19.3, user=root@6e7732a7b81b, date=20221129-18:59:09)" 2024-03-20T08:47:31.342 INFO:journalctl@ceph.node-exporter.a.smithi183.stdout:Mar 20 08:47:31 smithi183 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-node-exporter-a[62213]: ts=2024-03-20T08:47:31.195Z caller=diskstats_common.go:111 level=info collector=diskstats msg="Parsed flag --collector.diskstats.device-exclude" flag=^(ram|loop|fd|(h|s|v|xv)d[a-z]|nvme\d+n\d+p)\d+$ 2024-03-20T08:47:31.343 INFO:journalctl@ceph.node-exporter.a.smithi183.stdout:Mar 20 08:47:31 smithi183 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-node-exporter-a[62213]: ts=2024-03-20T08:47:31.195Z caller=diskstats_linux.go:264 level=error collector=diskstats msg="Failed to open directory, disabling udev device properties" path=/run/udev/data 2024-03-20T08:47:31.343 INFO:journalctl@ceph.node-exporter.a.smithi183.stdout:Mar 20 08:47:31 smithi183 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-node-exporter-a[62213]: ts=2024-03-20T08:47:31.195Z caller=filesystem_common.go:111 level=info collector=filesystem msg="Parsed flag --collector.filesystem.mount-points-exclude" flag=^/(dev|proc|run/credentials/.+|sys|var/lib/docker/.+|var/lib/containers/storage/.+)($|/) 2024-03-20T08:47:31.343 INFO:journalctl@ceph.node-exporter.a.smithi183.stdout:Mar 20 08:47:31 smithi183 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-node-exporter-a[62213]: ts=2024-03-20T08:47:31.195Z caller=filesystem_common.go:113 level=info collector=filesystem msg="Parsed flag --collector.filesystem.fs-types-exclude" flag=^(autofs|binfmt_misc|bpf|cgroup2?|configfs|debugfs|devpts|devtmpfs|fusectl|hugetlbfs|iso9660|mqueue|nsfs|overlay|proc|procfs|pstore|rpc_pipefs|securityfs|selinuxfs|squashfs|sysfs|tracefs)$ 2024-03-20T08:47:31.343 INFO:journalctl@ceph.node-exporter.a.smithi183.stdout:Mar 20 08:47:31 smithi183 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-node-exporter-a[62213]: ts=2024-03-20T08:47:31.196Z caller=node_exporter.go:110 level=info msg="Enabled collectors" 2024-03-20T08:47:31.343 INFO:journalctl@ceph.node-exporter.a.smithi183.stdout:Mar 20 08:47:31 smithi183 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-node-exporter-a[62213]: ts=2024-03-20T08:47:31.196Z caller=node_exporter.go:117 level=info collector=arp 2024-03-20T08:47:31.343 INFO:journalctl@ceph.node-exporter.a.smithi183.stdout:Mar 20 08:47:31 smithi183 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-node-exporter-a[62213]: ts=2024-03-20T08:47:31.196Z caller=node_exporter.go:117 level=info collector=bcache 2024-03-20T08:47:31.343 INFO:journalctl@ceph.node-exporter.a.smithi183.stdout:Mar 20 08:47:31 smithi183 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-node-exporter-a[62213]: ts=2024-03-20T08:47:31.196Z caller=node_exporter.go:117 level=info collector=bonding 2024-03-20T08:47:31.343 INFO:journalctl@ceph.node-exporter.a.smithi183.stdout:Mar 20 08:47:31 smithi183 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-node-exporter-a[62213]: ts=2024-03-20T08:47:31.196Z caller=node_exporter.go:117 level=info collector=btrfs 2024-03-20T08:47:31.343 INFO:journalctl@ceph.node-exporter.a.smithi183.stdout:Mar 20 08:47:31 smithi183 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-node-exporter-a[62213]: ts=2024-03-20T08:47:31.196Z caller=node_exporter.go:117 level=info collector=conntrack 2024-03-20T08:47:31.343 INFO:journalctl@ceph.node-exporter.a.smithi183.stdout:Mar 20 08:47:31 smithi183 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-node-exporter-a[62213]: ts=2024-03-20T08:47:31.196Z caller=node_exporter.go:117 level=info collector=cpu 2024-03-20T08:47:31.343 INFO:journalctl@ceph.node-exporter.a.smithi183.stdout:Mar 20 08:47:31 smithi183 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-node-exporter-a[62213]: ts=2024-03-20T08:47:31.196Z caller=node_exporter.go:117 level=info collector=cpufreq 2024-03-20T08:47:31.344 INFO:journalctl@ceph.node-exporter.a.smithi183.stdout:Mar 20 08:47:31 smithi183 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-node-exporter-a[62213]: ts=2024-03-20T08:47:31.196Z caller=node_exporter.go:117 level=info collector=diskstats 2024-03-20T08:47:31.344 INFO:journalctl@ceph.node-exporter.a.smithi183.stdout:Mar 20 08:47:31 smithi183 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-node-exporter-a[62213]: ts=2024-03-20T08:47:31.196Z caller=node_exporter.go:117 level=info collector=dmi 2024-03-20T08:47:31.344 INFO:journalctl@ceph.node-exporter.a.smithi183.stdout:Mar 20 08:47:31 smithi183 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-node-exporter-a[62213]: ts=2024-03-20T08:47:31.196Z caller=node_exporter.go:117 level=info collector=edac 2024-03-20T08:47:31.344 INFO:journalctl@ceph.node-exporter.a.smithi183.stdout:Mar 20 08:47:31 smithi183 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-node-exporter-a[62213]: ts=2024-03-20T08:47:31.196Z caller=node_exporter.go:117 level=info collector=entropy 2024-03-20T08:47:31.344 INFO:journalctl@ceph.node-exporter.a.smithi183.stdout:Mar 20 08:47:31 smithi183 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-node-exporter-a[62213]: ts=2024-03-20T08:47:31.196Z caller=node_exporter.go:117 level=info collector=fibrechannel 2024-03-20T08:47:31.344 INFO:journalctl@ceph.node-exporter.a.smithi183.stdout:Mar 20 08:47:31 smithi183 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-node-exporter-a[62213]: ts=2024-03-20T08:47:31.196Z caller=node_exporter.go:117 level=info collector=filefd 2024-03-20T08:47:31.344 INFO:journalctl@ceph.node-exporter.a.smithi183.stdout:Mar 20 08:47:31 smithi183 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-node-exporter-a[62213]: ts=2024-03-20T08:47:31.196Z caller=node_exporter.go:117 level=info collector=filesystem 2024-03-20T08:47:31.344 INFO:journalctl@ceph.node-exporter.a.smithi183.stdout:Mar 20 08:47:31 smithi183 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-node-exporter-a[62213]: ts=2024-03-20T08:47:31.196Z caller=node_exporter.go:117 level=info collector=hwmon 2024-03-20T08:47:31.344 INFO:journalctl@ceph.node-exporter.a.smithi183.stdout:Mar 20 08:47:31 smithi183 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-node-exporter-a[62213]: ts=2024-03-20T08:47:31.196Z caller=node_exporter.go:117 level=info collector=infiniband 2024-03-20T08:47:31.345 INFO:journalctl@ceph.node-exporter.a.smithi183.stdout:Mar 20 08:47:31 smithi183 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-node-exporter-a[62213]: ts=2024-03-20T08:47:31.196Z caller=node_exporter.go:117 level=info collector=ipvs 2024-03-20T08:47:31.345 INFO:journalctl@ceph.node-exporter.a.smithi183.stdout:Mar 20 08:47:31 smithi183 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-node-exporter-a[62213]: ts=2024-03-20T08:47:31.196Z caller=node_exporter.go:117 level=info collector=loadavg 2024-03-20T08:47:31.345 INFO:journalctl@ceph.node-exporter.a.smithi183.stdout:Mar 20 08:47:31 smithi183 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-node-exporter-a[62213]: ts=2024-03-20T08:47:31.196Z caller=node_exporter.go:117 level=info collector=mdadm 2024-03-20T08:47:31.345 INFO:journalctl@ceph.node-exporter.a.smithi183.stdout:Mar 20 08:47:31 smithi183 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-node-exporter-a[62213]: ts=2024-03-20T08:47:31.196Z caller=node_exporter.go:117 level=info collector=meminfo 2024-03-20T08:47:31.345 INFO:journalctl@ceph.node-exporter.a.smithi183.stdout:Mar 20 08:47:31 smithi183 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-node-exporter-a[62213]: ts=2024-03-20T08:47:31.196Z caller=node_exporter.go:117 level=info collector=netclass 2024-03-20T08:47:31.345 INFO:journalctl@ceph.node-exporter.a.smithi183.stdout:Mar 20 08:47:31 smithi183 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-node-exporter-a[62213]: ts=2024-03-20T08:47:31.196Z caller=node_exporter.go:117 level=info collector=netdev 2024-03-20T08:47:31.345 INFO:journalctl@ceph.node-exporter.a.smithi183.stdout:Mar 20 08:47:31 smithi183 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-node-exporter-a[62213]: ts=2024-03-20T08:47:31.196Z caller=node_exporter.go:117 level=info collector=netstat 2024-03-20T08:47:31.345 INFO:journalctl@ceph.node-exporter.a.smithi183.stdout:Mar 20 08:47:31 smithi183 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-node-exporter-a[62213]: ts=2024-03-20T08:47:31.196Z caller=node_exporter.go:117 level=info collector=nfs 2024-03-20T08:47:31.345 INFO:journalctl@ceph.node-exporter.a.smithi183.stdout:Mar 20 08:47:31 smithi183 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-node-exporter-a[62213]: ts=2024-03-20T08:47:31.196Z caller=node_exporter.go:117 level=info collector=nfsd 2024-03-20T08:47:31.345 INFO:journalctl@ceph.node-exporter.a.smithi183.stdout:Mar 20 08:47:31 smithi183 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-node-exporter-a[62213]: ts=2024-03-20T08:47:31.196Z caller=node_exporter.go:117 level=info collector=nvme 2024-03-20T08:47:31.345 INFO:journalctl@ceph.node-exporter.a.smithi183.stdout:Mar 20 08:47:31 smithi183 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-node-exporter-a[62213]: ts=2024-03-20T08:47:31.196Z caller=node_exporter.go:117 level=info collector=os 2024-03-20T08:47:31.345 INFO:journalctl@ceph.node-exporter.a.smithi183.stdout:Mar 20 08:47:31 smithi183 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-node-exporter-a[62213]: ts=2024-03-20T08:47:31.196Z caller=node_exporter.go:117 level=info collector=powersupplyclass 2024-03-20T08:47:31.345 INFO:journalctl@ceph.node-exporter.a.smithi183.stdout:Mar 20 08:47:31 smithi183 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-node-exporter-a[62213]: ts=2024-03-20T08:47:31.196Z caller=node_exporter.go:117 level=info collector=pressure 2024-03-20T08:47:31.345 INFO:journalctl@ceph.node-exporter.a.smithi183.stdout:Mar 20 08:47:31 smithi183 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-node-exporter-a[62213]: ts=2024-03-20T08:47:31.196Z caller=node_exporter.go:117 level=info collector=rapl 2024-03-20T08:47:31.346 INFO:journalctl@ceph.node-exporter.a.smithi183.stdout:Mar 20 08:47:31 smithi183 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-node-exporter-a[62213]: ts=2024-03-20T08:47:31.196Z caller=node_exporter.go:117 level=info collector=schedstat 2024-03-20T08:47:31.346 INFO:journalctl@ceph.node-exporter.a.smithi183.stdout:Mar 20 08:47:31 smithi183 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-node-exporter-a[62213]: ts=2024-03-20T08:47:31.196Z caller=node_exporter.go:117 level=info collector=selinux 2024-03-20T08:47:31.346 INFO:journalctl@ceph.node-exporter.a.smithi183.stdout:Mar 20 08:47:31 smithi183 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-node-exporter-a[62213]: ts=2024-03-20T08:47:31.196Z caller=node_exporter.go:117 level=info collector=sockstat 2024-03-20T08:47:31.346 INFO:journalctl@ceph.node-exporter.a.smithi183.stdout:Mar 20 08:47:31 smithi183 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-node-exporter-a[62213]: ts=2024-03-20T08:47:31.196Z caller=node_exporter.go:117 level=info collector=softnet 2024-03-20T08:47:31.346 INFO:journalctl@ceph.node-exporter.a.smithi183.stdout:Mar 20 08:47:31 smithi183 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-node-exporter-a[62213]: ts=2024-03-20T08:47:31.196Z caller=node_exporter.go:117 level=info collector=stat 2024-03-20T08:47:31.346 INFO:journalctl@ceph.node-exporter.a.smithi183.stdout:Mar 20 08:47:31 smithi183 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-node-exporter-a[62213]: ts=2024-03-20T08:47:31.196Z caller=node_exporter.go:117 level=info collector=tapestats 2024-03-20T08:47:31.346 INFO:journalctl@ceph.node-exporter.a.smithi183.stdout:Mar 20 08:47:31 smithi183 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-node-exporter-a[62213]: ts=2024-03-20T08:47:31.196Z caller=node_exporter.go:117 level=info collector=textfile 2024-03-20T08:47:31.346 INFO:journalctl@ceph.node-exporter.a.smithi183.stdout:Mar 20 08:47:31 smithi183 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-node-exporter-a[62213]: ts=2024-03-20T08:47:31.196Z caller=node_exporter.go:117 level=info collector=thermal_zone 2024-03-20T08:47:31.346 INFO:journalctl@ceph.node-exporter.a.smithi183.stdout:Mar 20 08:47:31 smithi183 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-node-exporter-a[62213]: ts=2024-03-20T08:47:31.196Z caller=node_exporter.go:117 level=info collector=time 2024-03-20T08:47:31.346 INFO:journalctl@ceph.node-exporter.a.smithi183.stdout:Mar 20 08:47:31 smithi183 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-node-exporter-a[62213]: ts=2024-03-20T08:47:31.196Z caller=node_exporter.go:117 level=info collector=udp_queues 2024-03-20T08:47:31.346 INFO:journalctl@ceph.node-exporter.a.smithi183.stdout:Mar 20 08:47:31 smithi183 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-node-exporter-a[62213]: ts=2024-03-20T08:47:31.196Z caller=node_exporter.go:117 level=info collector=uname 2024-03-20T08:47:31.346 INFO:journalctl@ceph.node-exporter.a.smithi183.stdout:Mar 20 08:47:31 smithi183 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-node-exporter-a[62213]: ts=2024-03-20T08:47:31.196Z caller=node_exporter.go:117 level=info collector=vmstat 2024-03-20T08:47:31.346 INFO:journalctl@ceph.node-exporter.a.smithi183.stdout:Mar 20 08:47:31 smithi183 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-node-exporter-a[62213]: ts=2024-03-20T08:47:31.196Z caller=node_exporter.go:117 level=info collector=xfs 2024-03-20T08:47:31.346 INFO:journalctl@ceph.node-exporter.a.smithi183.stdout:Mar 20 08:47:31 smithi183 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-node-exporter-a[62213]: ts=2024-03-20T08:47:31.196Z caller=node_exporter.go:117 level=info collector=zfs 2024-03-20T08:47:31.347 INFO:journalctl@ceph.node-exporter.a.smithi183.stdout:Mar 20 08:47:31 smithi183 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-node-exporter-a[62213]: ts=2024-03-20T08:47:31.197Z caller=tls_config.go:232 level=info msg="Listening on" address=[::]:9100 2024-03-20T08:47:31.347 INFO:journalctl@ceph.node-exporter.a.smithi183.stdout:Mar 20 08:47:31 smithi183 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-node-exporter-a[62213]: ts=2024-03-20T08:47:31.197Z caller=tls_config.go:235 level=info msg="TLS is disabled." http2=false address=[::]:9100 2024-03-20T08:47:31.347 INFO:journalctl@ceph.node-exporter.a.smithi183.stdout:Mar 20 08:47:31 smithi183 bash[61994]: 0dace97e4fc4562217de3d754934c883b3949f55529b51ef7f64abc4d49e3d5b 2024-03-20T08:47:31.347 INFO:journalctl@ceph.node-exporter.a.smithi183.stdout:Mar 20 08:47:31 smithi183 systemd[1]: Started Ceph node-exporter.a for 4c3f3eba-e695-11ee-95c9-87774f69a715. 2024-03-20T08:47:31.368 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/4c3f3eba-e695-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T08:47:31.388 INFO:journalctl@ceph.iscsi.iscsi.a.smithi194.stdout:Mar 20 08:47:30 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-iscsi-iscsi-a[47467]: debug there is no tcmu-runner data available 2024-03-20T08:47:31.934 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:47:31 smithi183 ceph-mon[25285]: osdmap e71: 8 total, 8 up, 8 in 2024-03-20T08:47:31.935 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:47:31 smithi183 ceph-mon[25285]: from='client.14559 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-20T08:47:31.935 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:47:31 smithi183 ceph-mon[25285]: from='mgr.14637 ' entity='mgr.y' 2024-03-20T08:47:31.935 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:47:31 smithi183 ceph-mon[25285]: from='mgr.14637 ' entity='mgr.y' 2024-03-20T08:47:31.935 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:47:31 smithi183 ceph-mon[25285]: from='mgr.14637 ' entity='mgr.y' 2024-03-20T08:47:31.935 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:47:31 smithi183 ceph-mon[25285]: Deploying daemon node-exporter.b on smithi194 2024-03-20T08:47:31.935 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:47:31 smithi183 ceph-mon[31860]: osdmap e71: 8 total, 8 up, 8 in 2024-03-20T08:47:31.935 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:47:31 smithi183 ceph-mon[31860]: from='client.14559 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-20T08:47:31.935 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:47:31 smithi183 ceph-mon[31860]: from='mgr.14637 ' entity='mgr.y' 2024-03-20T08:47:31.935 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:47:31 smithi183 ceph-mon[31860]: from='mgr.14637 ' entity='mgr.y' 2024-03-20T08:47:31.935 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:47:31 smithi183 ceph-mon[31860]: from='mgr.14637 ' entity='mgr.y' 2024-03-20T08:47:31.935 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:47:31 smithi183 ceph-mon[31860]: Deploying daemon node-exporter.b on smithi194 2024-03-20T08:47:32.036 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:47:31 smithi194 ceph-mon[27237]: osdmap e71: 8 total, 8 up, 8 in 2024-03-20T08:47:32.037 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:47:31 smithi194 ceph-mon[27237]: from='client.14559 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-20T08:47:32.037 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:47:31 smithi194 ceph-mon[27237]: from='mgr.14637 ' entity='mgr.y' 2024-03-20T08:47:32.037 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:47:31 smithi194 ceph-mon[27237]: from='mgr.14637 ' entity='mgr.y' 2024-03-20T08:47:32.037 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:47:31 smithi194 ceph-mon[27237]: from='mgr.14637 ' entity='mgr.y' 2024-03-20T08:47:32.037 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:47:31 smithi194 ceph-mon[27237]: Deploying daemon node-exporter.b on smithi194 2024-03-20T08:47:32.639 INFO:journalctl@ceph.node-exporter.b.smithi194.stdout:Mar 20 08:47:32 smithi194 bash[53013]: Trying to pull quay.io/prometheus/node-exporter:v1.5.0... 2024-03-20T08:47:32.840 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:47:32 smithi183 ceph-mon[25285]: pgmap v36: 132 pgs: 3 peering, 129 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-20T08:47:32.841 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:47:32 smithi183 ceph-mon[25285]: Health check failed: Reduced data availability: 3 pgs peering (PG_AVAILABILITY) 2024-03-20T08:47:32.841 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:47:32 smithi183 ceph-mon[31860]: pgmap v36: 132 pgs: 3 peering, 129 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-20T08:47:32.841 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:47:32 smithi183 ceph-mon[31860]: Health check failed: Reduced data availability: 3 pgs peering (PG_AVAILABILITY) 2024-03-20T08:47:33.139 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:47:32 smithi194 ceph-mon[27237]: pgmap v36: 132 pgs: 3 peering, 129 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-20T08:47:33.139 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:47:32 smithi194 ceph-mon[27237]: Health check failed: Reduced data availability: 3 pgs peering (PG_AVAILABILITY) 2024-03-20T08:47:33.139 INFO:journalctl@ceph.node-exporter.b.smithi194.stdout:Mar 20 08:47:32 smithi194 bash[53013]: Getting image source signatures 2024-03-20T08:47:33.139 INFO:journalctl@ceph.node-exporter.b.smithi194.stdout:Mar 20 08:47:32 smithi194 bash[53013]: Copying blob sha256:c0e87333d3805a7b7e1c25783dd25f27da0fe70a8e262eee7ed190c8dc843fa7 2024-03-20T08:47:33.140 INFO:journalctl@ceph.node-exporter.b.smithi194.stdout:Mar 20 08:47:32 smithi194 bash[53013]: Copying blob sha256:22b70bddd3acadc892fca4c2af4260629bfda5dfd11ebc106a93ce24e752b5ed 2024-03-20T08:47:33.140 INFO:journalctl@ceph.node-exporter.b.smithi194.stdout:Mar 20 08:47:32 smithi194 bash[53013]: Copying blob sha256:5c12815fee558b157dee7f7509dedbaba0a8379098858a65ec869e1f1526ea0c 2024-03-20T08:47:34.332 INFO:teuthology.orchestra.run.smithi183.stdout:Backend: cephadm 2024-03-20T08:47:34.332 INFO:teuthology.orchestra.run.smithi183.stdout:Available: Yes 2024-03-20T08:47:34.332 INFO:teuthology.orchestra.run.smithi183.stdout:Paused: No 2024-03-20T08:47:34.722 INFO:journalctl@ceph.node-exporter.b.smithi194.stdout:Mar 20 08:47:34 smithi194 bash[53013]: Copying config sha256:0da6a335fe1356545476b749c68f022c897de3a2139e8f0054f6937349ee2b83 2024-03-20T08:47:34.723 INFO:journalctl@ceph.node-exporter.b.smithi194.stdout:Mar 20 08:47:34 smithi194 bash[53013]: Writing manifest to image destination 2024-03-20T08:47:34.916 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:47:34 smithi183 ceph-mon[25285]: pgmap v37: 132 pgs: 3 peering, 129 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-20T08:47:34.917 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:47:34 smithi183 ceph-mon[25285]: from='client.14799 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-20T08:47:34.917 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:47:34 smithi183 ceph-mon[31860]: pgmap v37: 132 pgs: 3 peering, 129 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-20T08:47:34.917 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:47:34 smithi183 ceph-mon[31860]: from='client.14799 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-20T08:47:35.057 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:47:34 smithi194 ceph-mon[27237]: pgmap v37: 132 pgs: 3 peering, 129 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-20T08:47:35.057 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:47:34 smithi194 ceph-mon[27237]: from='client.14799 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-20T08:47:35.058 INFO:journalctl@ceph.node-exporter.b.smithi194.stdout:Mar 20 08:47:34 smithi194 podman[53013]: 2024-03-20 08:47:34.865232695 +0000 UTC m=+2.408166158 image pull 0da6a335fe1356545476b749c68f022c897de3a2139e8f0054f6937349ee2b83 quay.io/prometheus/node-exporter:v1.5.0 2024-03-20T08:47:35.058 INFO:journalctl@ceph.node-exporter.b.smithi194.stdout:Mar 20 08:47:34 smithi194 podman[53013]: 2024-03-20 08:47:34.963709413 +0000 UTC m=+2.506642876 container create fed6ffe6922c5f9411b6b1993e01d015efccce0cb8bb290aa1c7240dc831eea6 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-node-exporter-b, maintainer=The Prometheus Authors ) 2024-03-20T08:47:35.389 INFO:journalctl@ceph.node-exporter.b.smithi194.stdout:Mar 20 08:47:35 smithi194 podman[53013]: 2024-03-20 08:47:35.190758562 +0000 UTC m=+2.733692025 container init fed6ffe6922c5f9411b6b1993e01d015efccce0cb8bb290aa1c7240dc831eea6 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-node-exporter-b, maintainer=The Prometheus Authors ) 2024-03-20T08:47:35.389 INFO:journalctl@ceph.node-exporter.b.smithi194.stdout:Mar 20 08:47:35 smithi194 podman[53013]: 2024-03-20 08:47:35.193643103 +0000 UTC m=+2.736576566 container start fed6ffe6922c5f9411b6b1993e01d015efccce0cb8bb290aa1c7240dc831eea6 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-node-exporter-b, maintainer=The Prometheus Authors ) 2024-03-20T08:47:35.389 INFO:journalctl@ceph.node-exporter.b.smithi194.stdout:Mar 20 08:47:35 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-node-exporter-b[53065]: ts=2024-03-20T08:47:35.198Z caller=node_exporter.go:180 level=info msg="Starting node_exporter" version="(version=1.5.0, branch=HEAD, revision=1b48970ffcf5630534fb00bb0687d73c66d1c959)" 2024-03-20T08:47:35.389 INFO:journalctl@ceph.node-exporter.b.smithi194.stdout:Mar 20 08:47:35 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-node-exporter-b[53065]: ts=2024-03-20T08:47:35.198Z caller=node_exporter.go:181 level=info msg="Build context" build_context="(go=go1.19.3, user=root@6e7732a7b81b, date=20221129-18:59:09)" 2024-03-20T08:47:35.389 INFO:journalctl@ceph.node-exporter.b.smithi194.stdout:Mar 20 08:47:35 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-node-exporter-b[53065]: ts=2024-03-20T08:47:35.198Z caller=diskstats_common.go:111 level=info collector=diskstats msg="Parsed flag --collector.diskstats.device-exclude" flag=^(ram|loop|fd|(h|s|v|xv)d[a-z]|nvme\d+n\d+p)\d+$ 2024-03-20T08:47:35.389 INFO:journalctl@ceph.node-exporter.b.smithi194.stdout:Mar 20 08:47:35 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-node-exporter-b[53065]: ts=2024-03-20T08:47:35.198Z caller=diskstats_linux.go:264 level=error collector=diskstats msg="Failed to open directory, disabling udev device properties" path=/run/udev/data 2024-03-20T08:47:35.389 INFO:journalctl@ceph.node-exporter.b.smithi194.stdout:Mar 20 08:47:35 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-node-exporter-b[53065]: ts=2024-03-20T08:47:35.198Z caller=filesystem_common.go:111 level=info collector=filesystem msg="Parsed flag --collector.filesystem.mount-points-exclude" flag=^/(dev|proc|run/credentials/.+|sys|var/lib/docker/.+|var/lib/containers/storage/.+)($|/) 2024-03-20T08:47:35.389 INFO:journalctl@ceph.node-exporter.b.smithi194.stdout:Mar 20 08:47:35 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-node-exporter-b[53065]: ts=2024-03-20T08:47:35.198Z caller=filesystem_common.go:113 level=info collector=filesystem msg="Parsed flag --collector.filesystem.fs-types-exclude" flag=^(autofs|binfmt_misc|bpf|cgroup2?|configfs|debugfs|devpts|devtmpfs|fusectl|hugetlbfs|iso9660|mqueue|nsfs|overlay|proc|procfs|pstore|rpc_pipefs|securityfs|selinuxfs|squashfs|sysfs|tracefs)$ 2024-03-20T08:47:35.389 INFO:journalctl@ceph.node-exporter.b.smithi194.stdout:Mar 20 08:47:35 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-node-exporter-b[53065]: ts=2024-03-20T08:47:35.199Z caller=node_exporter.go:110 level=info msg="Enabled collectors" 2024-03-20T08:47:35.390 INFO:journalctl@ceph.node-exporter.b.smithi194.stdout:Mar 20 08:47:35 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-node-exporter-b[53065]: ts=2024-03-20T08:47:35.199Z caller=node_exporter.go:117 level=info collector=arp 2024-03-20T08:47:35.390 INFO:journalctl@ceph.node-exporter.b.smithi194.stdout:Mar 20 08:47:35 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-node-exporter-b[53065]: ts=2024-03-20T08:47:35.199Z caller=node_exporter.go:117 level=info collector=bcache 2024-03-20T08:47:35.390 INFO:journalctl@ceph.node-exporter.b.smithi194.stdout:Mar 20 08:47:35 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-node-exporter-b[53065]: ts=2024-03-20T08:47:35.199Z caller=node_exporter.go:117 level=info collector=bonding 2024-03-20T08:47:35.390 INFO:journalctl@ceph.node-exporter.b.smithi194.stdout:Mar 20 08:47:35 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-node-exporter-b[53065]: ts=2024-03-20T08:47:35.199Z caller=node_exporter.go:117 level=info collector=btrfs 2024-03-20T08:47:35.390 INFO:journalctl@ceph.node-exporter.b.smithi194.stdout:Mar 20 08:47:35 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-node-exporter-b[53065]: ts=2024-03-20T08:47:35.199Z caller=node_exporter.go:117 level=info collector=conntrack 2024-03-20T08:47:35.390 INFO:journalctl@ceph.node-exporter.b.smithi194.stdout:Mar 20 08:47:35 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-node-exporter-b[53065]: ts=2024-03-20T08:47:35.199Z caller=node_exporter.go:117 level=info collector=cpu 2024-03-20T08:47:35.390 INFO:journalctl@ceph.node-exporter.b.smithi194.stdout:Mar 20 08:47:35 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-node-exporter-b[53065]: ts=2024-03-20T08:47:35.199Z caller=node_exporter.go:117 level=info collector=cpufreq 2024-03-20T08:47:35.391 INFO:journalctl@ceph.node-exporter.b.smithi194.stdout:Mar 20 08:47:35 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-node-exporter-b[53065]: ts=2024-03-20T08:47:35.199Z caller=node_exporter.go:117 level=info collector=diskstats 2024-03-20T08:47:35.391 INFO:journalctl@ceph.node-exporter.b.smithi194.stdout:Mar 20 08:47:35 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-node-exporter-b[53065]: ts=2024-03-20T08:47:35.199Z caller=node_exporter.go:117 level=info collector=dmi 2024-03-20T08:47:35.391 INFO:journalctl@ceph.node-exporter.b.smithi194.stdout:Mar 20 08:47:35 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-node-exporter-b[53065]: ts=2024-03-20T08:47:35.199Z caller=node_exporter.go:117 level=info collector=edac 2024-03-20T08:47:35.391 INFO:journalctl@ceph.node-exporter.b.smithi194.stdout:Mar 20 08:47:35 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-node-exporter-b[53065]: ts=2024-03-20T08:47:35.199Z caller=node_exporter.go:117 level=info collector=entropy 2024-03-20T08:47:35.391 INFO:journalctl@ceph.node-exporter.b.smithi194.stdout:Mar 20 08:47:35 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-node-exporter-b[53065]: ts=2024-03-20T08:47:35.199Z caller=node_exporter.go:117 level=info collector=fibrechannel 2024-03-20T08:47:35.391 INFO:journalctl@ceph.node-exporter.b.smithi194.stdout:Mar 20 08:47:35 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-node-exporter-b[53065]: ts=2024-03-20T08:47:35.199Z caller=node_exporter.go:117 level=info collector=filefd 2024-03-20T08:47:35.391 INFO:journalctl@ceph.node-exporter.b.smithi194.stdout:Mar 20 08:47:35 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-node-exporter-b[53065]: ts=2024-03-20T08:47:35.199Z caller=node_exporter.go:117 level=info collector=filesystem 2024-03-20T08:47:35.391 INFO:journalctl@ceph.node-exporter.b.smithi194.stdout:Mar 20 08:47:35 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-node-exporter-b[53065]: ts=2024-03-20T08:47:35.199Z caller=node_exporter.go:117 level=info collector=hwmon 2024-03-20T08:47:35.391 INFO:journalctl@ceph.node-exporter.b.smithi194.stdout:Mar 20 08:47:35 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-node-exporter-b[53065]: ts=2024-03-20T08:47:35.199Z caller=node_exporter.go:117 level=info collector=infiniband 2024-03-20T08:47:35.392 INFO:journalctl@ceph.node-exporter.b.smithi194.stdout:Mar 20 08:47:35 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-node-exporter-b[53065]: ts=2024-03-20T08:47:35.199Z caller=node_exporter.go:117 level=info collector=ipvs 2024-03-20T08:47:35.392 INFO:journalctl@ceph.node-exporter.b.smithi194.stdout:Mar 20 08:47:35 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-node-exporter-b[53065]: ts=2024-03-20T08:47:35.199Z caller=node_exporter.go:117 level=info collector=loadavg 2024-03-20T08:47:35.392 INFO:journalctl@ceph.node-exporter.b.smithi194.stdout:Mar 20 08:47:35 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-node-exporter-b[53065]: ts=2024-03-20T08:47:35.199Z caller=node_exporter.go:117 level=info collector=mdadm 2024-03-20T08:47:35.392 INFO:journalctl@ceph.node-exporter.b.smithi194.stdout:Mar 20 08:47:35 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-node-exporter-b[53065]: ts=2024-03-20T08:47:35.199Z caller=node_exporter.go:117 level=info collector=meminfo 2024-03-20T08:47:35.392 INFO:journalctl@ceph.node-exporter.b.smithi194.stdout:Mar 20 08:47:35 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-node-exporter-b[53065]: ts=2024-03-20T08:47:35.199Z caller=node_exporter.go:117 level=info collector=netclass 2024-03-20T08:47:35.392 INFO:journalctl@ceph.node-exporter.b.smithi194.stdout:Mar 20 08:47:35 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-node-exporter-b[53065]: ts=2024-03-20T08:47:35.199Z caller=node_exporter.go:117 level=info collector=netdev 2024-03-20T08:47:35.392 INFO:journalctl@ceph.node-exporter.b.smithi194.stdout:Mar 20 08:47:35 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-node-exporter-b[53065]: ts=2024-03-20T08:47:35.199Z caller=node_exporter.go:117 level=info collector=netstat 2024-03-20T08:47:35.392 INFO:journalctl@ceph.node-exporter.b.smithi194.stdout:Mar 20 08:47:35 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-node-exporter-b[53065]: ts=2024-03-20T08:47:35.199Z caller=node_exporter.go:117 level=info collector=nfs 2024-03-20T08:47:35.392 INFO:journalctl@ceph.node-exporter.b.smithi194.stdout:Mar 20 08:47:35 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-node-exporter-b[53065]: ts=2024-03-20T08:47:35.199Z caller=node_exporter.go:117 level=info collector=nfsd 2024-03-20T08:47:35.392 INFO:journalctl@ceph.node-exporter.b.smithi194.stdout:Mar 20 08:47:35 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-node-exporter-b[53065]: ts=2024-03-20T08:47:35.199Z caller=node_exporter.go:117 level=info collector=nvme 2024-03-20T08:47:35.392 INFO:journalctl@ceph.node-exporter.b.smithi194.stdout:Mar 20 08:47:35 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-node-exporter-b[53065]: ts=2024-03-20T08:47:35.199Z caller=node_exporter.go:117 level=info collector=os 2024-03-20T08:47:35.392 INFO:journalctl@ceph.node-exporter.b.smithi194.stdout:Mar 20 08:47:35 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-node-exporter-b[53065]: ts=2024-03-20T08:47:35.199Z caller=node_exporter.go:117 level=info collector=powersupplyclass 2024-03-20T08:47:35.393 INFO:journalctl@ceph.node-exporter.b.smithi194.stdout:Mar 20 08:47:35 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-node-exporter-b[53065]: ts=2024-03-20T08:47:35.199Z caller=node_exporter.go:117 level=info collector=pressure 2024-03-20T08:47:35.393 INFO:journalctl@ceph.node-exporter.b.smithi194.stdout:Mar 20 08:47:35 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-node-exporter-b[53065]: ts=2024-03-20T08:47:35.199Z caller=node_exporter.go:117 level=info collector=rapl 2024-03-20T08:47:35.393 INFO:journalctl@ceph.node-exporter.b.smithi194.stdout:Mar 20 08:47:35 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-node-exporter-b[53065]: ts=2024-03-20T08:47:35.199Z caller=node_exporter.go:117 level=info collector=schedstat 2024-03-20T08:47:35.393 INFO:journalctl@ceph.node-exporter.b.smithi194.stdout:Mar 20 08:47:35 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-node-exporter-b[53065]: ts=2024-03-20T08:47:35.199Z caller=node_exporter.go:117 level=info collector=selinux 2024-03-20T08:47:35.393 INFO:journalctl@ceph.node-exporter.b.smithi194.stdout:Mar 20 08:47:35 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-node-exporter-b[53065]: ts=2024-03-20T08:47:35.199Z caller=node_exporter.go:117 level=info collector=sockstat 2024-03-20T08:47:35.393 INFO:journalctl@ceph.node-exporter.b.smithi194.stdout:Mar 20 08:47:35 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-node-exporter-b[53065]: ts=2024-03-20T08:47:35.199Z caller=node_exporter.go:117 level=info collector=softnet 2024-03-20T08:47:35.393 INFO:journalctl@ceph.node-exporter.b.smithi194.stdout:Mar 20 08:47:35 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-node-exporter-b[53065]: ts=2024-03-20T08:47:35.199Z caller=node_exporter.go:117 level=info collector=stat 2024-03-20T08:47:35.393 INFO:journalctl@ceph.node-exporter.b.smithi194.stdout:Mar 20 08:47:35 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-node-exporter-b[53065]: ts=2024-03-20T08:47:35.199Z caller=node_exporter.go:117 level=info collector=tapestats 2024-03-20T08:47:35.393 INFO:journalctl@ceph.node-exporter.b.smithi194.stdout:Mar 20 08:47:35 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-node-exporter-b[53065]: ts=2024-03-20T08:47:35.199Z caller=node_exporter.go:117 level=info collector=textfile 2024-03-20T08:47:35.394 INFO:journalctl@ceph.node-exporter.b.smithi194.stdout:Mar 20 08:47:35 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-node-exporter-b[53065]: ts=2024-03-20T08:47:35.199Z caller=node_exporter.go:117 level=info collector=thermal_zone 2024-03-20T08:47:35.394 INFO:journalctl@ceph.node-exporter.b.smithi194.stdout:Mar 20 08:47:35 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-node-exporter-b[53065]: ts=2024-03-20T08:47:35.199Z caller=node_exporter.go:117 level=info collector=time 2024-03-20T08:47:35.394 INFO:journalctl@ceph.node-exporter.b.smithi194.stdout:Mar 20 08:47:35 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-node-exporter-b[53065]: ts=2024-03-20T08:47:35.199Z caller=node_exporter.go:117 level=info collector=udp_queues 2024-03-20T08:47:35.394 INFO:journalctl@ceph.node-exporter.b.smithi194.stdout:Mar 20 08:47:35 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-node-exporter-b[53065]: ts=2024-03-20T08:47:35.199Z caller=node_exporter.go:117 level=info collector=uname 2024-03-20T08:47:35.394 INFO:journalctl@ceph.node-exporter.b.smithi194.stdout:Mar 20 08:47:35 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-node-exporter-b[53065]: ts=2024-03-20T08:47:35.199Z caller=node_exporter.go:117 level=info collector=vmstat 2024-03-20T08:47:35.394 INFO:journalctl@ceph.node-exporter.b.smithi194.stdout:Mar 20 08:47:35 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-node-exporter-b[53065]: ts=2024-03-20T08:47:35.199Z caller=node_exporter.go:117 level=info collector=xfs 2024-03-20T08:47:35.394 INFO:journalctl@ceph.node-exporter.b.smithi194.stdout:Mar 20 08:47:35 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-node-exporter-b[53065]: ts=2024-03-20T08:47:35.199Z caller=node_exporter.go:117 level=info collector=zfs 2024-03-20T08:47:35.394 INFO:journalctl@ceph.node-exporter.b.smithi194.stdout:Mar 20 08:47:35 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-node-exporter-b[53065]: ts=2024-03-20T08:47:35.199Z caller=tls_config.go:232 level=info msg="Listening on" address=[::]:9100 2024-03-20T08:47:35.394 INFO:journalctl@ceph.node-exporter.b.smithi194.stdout:Mar 20 08:47:35 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-node-exporter-b[53065]: ts=2024-03-20T08:47:35.199Z caller=tls_config.go:235 level=info msg="TLS is disabled." http2=false address=[::]:9100 2024-03-20T08:47:35.394 INFO:journalctl@ceph.node-exporter.b.smithi194.stdout:Mar 20 08:47:35 smithi194 bash[53013]: fed6ffe6922c5f9411b6b1993e01d015efccce0cb8bb290aa1c7240dc831eea6 2024-03-20T08:47:35.394 INFO:journalctl@ceph.node-exporter.b.smithi194.stdout:Mar 20 08:47:35 smithi194 systemd[1]: Started Ceph node-exporter.b for 4c3f3eba-e695-11ee-95c9-87774f69a715. 2024-03-20T08:47:35.445 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4c3f3eba-e695-11ee-95c9-87774f69a715 -- bash -c 'ceph orch ps' 2024-03-20T08:47:35.761 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/4c3f3eba-e695-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T08:47:36.388 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:47:36 smithi194 ceph-mon[27237]: from='mgr.14637 ' entity='mgr.y' 2024-03-20T08:47:36.389 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:47:36 smithi194 ceph-mon[27237]: from='mgr.14637 ' entity='mgr.y' 2024-03-20T08:47:36.389 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:47:36 smithi194 ceph-mon[27237]: from='mgr.14637 ' entity='mgr.y' 2024-03-20T08:47:36.389 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:47:36 smithi194 ceph-mon[27237]: from='mgr.14637 ' entity='mgr.y' 2024-03-20T08:47:36.389 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:47:36 smithi194 ceph-mon[27237]: from='mgr.14637 ' entity='mgr.y' 2024-03-20T08:47:36.389 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:47:36 smithi194 ceph-mon[27237]: from='mgr.14637 172.21.15.183:0/885080955' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-20T08:47:36.389 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:47:36 smithi194 ceph-mon[27237]: from='mgr.14637 172.21.15.183:0/885080955' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.agent.smithi183", "caps": []}]: dispatch 2024-03-20T08:47:36.389 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:47:36 smithi194 ceph-mon[27237]: from='mgr.14637 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.agent.smithi183", "caps": []}]: dispatch 2024-03-20T08:47:36.389 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:47:36 smithi194 ceph-mon[27237]: from='mgr.14637 172.21.15.183:0/885080955' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.agent.smithi194", "caps": []}]: dispatch 2024-03-20T08:47:36.389 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:47:36 smithi194 ceph-mon[27237]: from='mgr.14637 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.agent.smithi194", "caps": []}]: dispatch 2024-03-20T08:47:36.418 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:47:36 smithi183 ceph-mon[31860]: from='mgr.14637 ' entity='mgr.y' 2024-03-20T08:47:36.418 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:47:36 smithi183 ceph-mon[31860]: from='mgr.14637 ' entity='mgr.y' 2024-03-20T08:47:36.418 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:47:36 smithi183 ceph-mon[31860]: from='mgr.14637 ' entity='mgr.y' 2024-03-20T08:47:36.418 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:47:36 smithi183 ceph-mon[31860]: from='mgr.14637 ' entity='mgr.y' 2024-03-20T08:47:36.418 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:47:36 smithi183 ceph-mon[31860]: from='mgr.14637 ' entity='mgr.y' 2024-03-20T08:47:36.418 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:47:36 smithi183 ceph-mon[31860]: from='mgr.14637 172.21.15.183:0/885080955' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-20T08:47:36.419 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:47:36 smithi183 ceph-mon[31860]: from='mgr.14637 172.21.15.183:0/885080955' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.agent.smithi183", "caps": []}]: dispatch 2024-03-20T08:47:36.419 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:47:36 smithi183 ceph-mon[31860]: from='mgr.14637 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.agent.smithi183", "caps": []}]: dispatch 2024-03-20T08:47:36.419 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:47:36 smithi183 ceph-mon[31860]: from='mgr.14637 172.21.15.183:0/885080955' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.agent.smithi194", "caps": []}]: dispatch 2024-03-20T08:47:36.419 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:47:36 smithi183 ceph-mon[31860]: from='mgr.14637 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.agent.smithi194", "caps": []}]: dispatch 2024-03-20T08:47:36.419 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:47:36 smithi183 ceph-mon[25285]: from='mgr.14637 ' entity='mgr.y' 2024-03-20T08:47:36.419 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:47:36 smithi183 ceph-mon[25285]: from='mgr.14637 ' entity='mgr.y' 2024-03-20T08:47:36.419 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:47:36 smithi183 ceph-mon[25285]: from='mgr.14637 ' entity='mgr.y' 2024-03-20T08:47:36.419 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:47:36 smithi183 ceph-mon[25285]: from='mgr.14637 ' entity='mgr.y' 2024-03-20T08:47:36.419 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:47:36 smithi183 ceph-mon[25285]: from='mgr.14637 ' entity='mgr.y' 2024-03-20T08:47:36.419 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:47:36 smithi183 ceph-mon[25285]: from='mgr.14637 172.21.15.183:0/885080955' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-20T08:47:36.419 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:47:36 smithi183 ceph-mon[25285]: from='mgr.14637 172.21.15.183:0/885080955' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.agent.smithi183", "caps": []}]: dispatch 2024-03-20T08:47:36.420 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:47:36 smithi183 ceph-mon[25285]: from='mgr.14637 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.agent.smithi183", "caps": []}]: dispatch 2024-03-20T08:47:36.420 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:47:36 smithi183 ceph-mon[25285]: from='mgr.14637 172.21.15.183:0/885080955' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.agent.smithi194", "caps": []}]: dispatch 2024-03-20T08:47:36.420 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:47:36 smithi183 ceph-mon[25285]: from='mgr.14637 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.agent.smithi194", "caps": []}]: dispatch 2024-03-20T08:47:37.192 INFO:journalctl@ceph.mgr.y.smithi183.stdout:Mar 20 08:47:36 smithi183 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-mgr-y[25513]: ::ffff:172.21.15.194 - - [20/Mar/2024:08:47:36] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-20T08:47:37.565 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:47:37 smithi194 ceph-mon[27237]: pgmap v38: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 27 B/s, 1 objects/s recovering 2024-03-20T08:47:37.565 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:47:37 smithi194 ceph-mon[27237]: Reconfiguring daemon agent.smithi194 on smithi194 2024-03-20T08:47:37.565 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:47:37 smithi194 ceph-mon[27237]: Reconfiguring daemon agent.smithi183 on smithi183 2024-03-20T08:47:37.565 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:47:37 smithi194 ceph-mon[27237]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 3 pgs peering) 2024-03-20T08:47:37.565 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:47:37 smithi194 ceph-mon[27237]: Cluster is now healthy 2024-03-20T08:47:37.566 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:47:37 smithi194 ceph-mon[27237]: from='mgr.14637 ' entity='mgr.y' 2024-03-20T08:47:37.704 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:47:37 smithi183 ceph-mon[25285]: pgmap v38: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 27 B/s, 1 objects/s recovering 2024-03-20T08:47:37.704 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:47:37 smithi183 ceph-mon[25285]: Reconfiguring daemon agent.smithi194 on smithi194 2024-03-20T08:47:37.704 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:47:37 smithi183 ceph-mon[25285]: Reconfiguring daemon agent.smithi183 on smithi183 2024-03-20T08:47:37.704 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:47:37 smithi183 ceph-mon[25285]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 3 pgs peering) 2024-03-20T08:47:37.705 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:47:37 smithi183 ceph-mon[25285]: Cluster is now healthy 2024-03-20T08:47:37.705 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:47:37 smithi183 ceph-mon[25285]: from='mgr.14637 ' entity='mgr.y' 2024-03-20T08:47:37.705 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:47:37 smithi183 ceph-mon[31860]: pgmap v38: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 27 B/s, 1 objects/s recovering 2024-03-20T08:47:37.705 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:47:37 smithi183 ceph-mon[31860]: Reconfiguring daemon agent.smithi194 on smithi194 2024-03-20T08:47:37.705 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:47:37 smithi183 ceph-mon[31860]: Reconfiguring daemon agent.smithi183 on smithi183 2024-03-20T08:47:37.705 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:47:37 smithi183 ceph-mon[31860]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 3 pgs peering) 2024-03-20T08:47:37.705 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:47:37 smithi183 ceph-mon[31860]: Cluster is now healthy 2024-03-20T08:47:37.705 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:47:37 smithi183 ceph-mon[31860]: from='mgr.14637 ' entity='mgr.y' 2024-03-20T08:47:38.667 INFO:teuthology.orchestra.run.smithi183.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-20T08:47:38.667 INFO:teuthology.orchestra.run.smithi183.stdout:agent.smithi183 smithi183 running 86s ago 6m - - 2024-03-20T08:47:38.667 INFO:teuthology.orchestra.run.smithi183.stdout:agent.smithi194 smithi194 starting - - - - 2024-03-20T08:47:38.667 INFO:teuthology.orchestra.run.smithi183.stdout:alertmanager.a smithi183 *:9093,9094 starting - - - - 2024-03-20T08:47:38.667 INFO:teuthology.orchestra.run.smithi183.stdout:grafana.a smithi194 *:3000 starting - - - - 2024-03-20T08:47:38.667 INFO:teuthology.orchestra.run.smithi183.stdout:iscsi.iscsi.a smithi194 *:5000 running (98s) 91s ago 97s - - 3.7 9ceba81fb2c8 589372ff43ad 2024-03-20T08:47:38.667 INFO:teuthology.orchestra.run.smithi183.stdout:mgr.x smithi194 *:8443,8765 running (5m) 91s ago 5m - - 19.0.0-2153-ge7e80a6a 9ceba81fb2c8 625764fcda26 2024-03-20T08:47:38.667 INFO:teuthology.orchestra.run.smithi183.stdout:mgr.y smithi183 *:9283,8765 running (7m) 86s ago 7m 509M - 19.0.0-2153-ge7e80a6a 9ceba81fb2c8 33072c5d6210 2024-03-20T08:47:38.667 INFO:teuthology.orchestra.run.smithi183.stdout:mon.a smithi183 running (7m) 86s ago 7m 44.9M 2048M 19.0.0-2153-ge7e80a6a 9ceba81fb2c8 8c5d9ae3ddbe 2024-03-20T08:47:38.667 INFO:teuthology.orchestra.run.smithi183.stdout:mon.b smithi194 running (6m) 91s ago 6m - 2048M 19.0.0-2153-ge7e80a6a 9ceba81fb2c8 da6a20d027a9 2024-03-20T08:47:38.667 INFO:teuthology.orchestra.run.smithi183.stdout:mon.c smithi183 running (5m) 86s ago 5m 39.1M 2048M 19.0.0-2153-ge7e80a6a 9ceba81fb2c8 c5f899930acf 2024-03-20T08:47:38.667 INFO:teuthology.orchestra.run.smithi183.stdout:node-exporter.a smithi183 *:9100 starting - - - - 2024-03-20T08:47:38.668 INFO:teuthology.orchestra.run.smithi183.stdout:node-exporter.b smithi194 *:9100 starting - - - - 2024-03-20T08:47:38.668 INFO:teuthology.orchestra.run.smithi183.stdout:osd.0 smithi183 running (5m) 86s ago 5m 46.4M 4096M 19.0.0-2153-ge7e80a6a 9ceba81fb2c8 e86ec9850e30 2024-03-20T08:47:38.668 INFO:teuthology.orchestra.run.smithi183.stdout:osd.1 smithi183 running (4m) 86s ago 4m 52.7M 4096M 19.0.0-2153-ge7e80a6a 9ceba81fb2c8 93c340a16ba3 2024-03-20T08:47:38.668 INFO:teuthology.orchestra.run.smithi183.stdout:osd.2 smithi183 running (4m) 86s ago 4m 44.6M 4096M 19.0.0-2153-ge7e80a6a 9ceba81fb2c8 db171b2995f2 2024-03-20T08:47:38.668 INFO:teuthology.orchestra.run.smithi183.stdout:osd.3 smithi183 running (3m) 86s ago 3m 51.6M 4096M 19.0.0-2153-ge7e80a6a 9ceba81fb2c8 77e843f75ce4 2024-03-20T08:47:38.668 INFO:teuthology.orchestra.run.smithi183.stdout:osd.4 smithi194 running (3m) 91s ago 3m - 4096M 19.0.0-2153-ge7e80a6a 9ceba81fb2c8 aac22f466d47 2024-03-20T08:47:38.668 INFO:teuthology.orchestra.run.smithi183.stdout:osd.5 smithi194 running (3m) 91s ago 3m - 4096M 19.0.0-2153-ge7e80a6a 9ceba81fb2c8 234c30f1c954 2024-03-20T08:47:38.668 INFO:teuthology.orchestra.run.smithi183.stdout:osd.6 smithi194 running (2m) 91s ago 2m - 4096M 19.0.0-2153-ge7e80a6a 9ceba81fb2c8 9a38fbc0b31e 2024-03-20T08:47:38.668 INFO:teuthology.orchestra.run.smithi183.stdout:osd.7 smithi194 running (2m) 91s ago 2m - 4096M 19.0.0-2153-ge7e80a6a 9ceba81fb2c8 007108f102c2 2024-03-20T08:47:38.668 INFO:teuthology.orchestra.run.smithi183.stdout:prometheus.a smithi194 *:9095 starting - - - - 2024-03-20T08:47:38.668 INFO:teuthology.orchestra.run.smithi183.stdout:rgw.foo.a smithi183 *:80 running (110s) 86s ago 110s 73.4M - 19.0.0-2153-ge7e80a6a 9ceba81fb2c8 b6a7f433ee92 2024-03-20T08:47:39.388 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:47:39 smithi194 ceph-mon[27237]: pgmap v39: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 27 B/s, 1 objects/s recovering 2024-03-20T08:47:39.454 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:47:39 smithi183 ceph-mon[25285]: pgmap v39: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 27 B/s, 1 objects/s recovering 2024-03-20T08:47:39.455 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:47:39 smithi183 ceph-mon[31860]: pgmap v39: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 27 B/s, 1 objects/s recovering 2024-03-20T08:47:39.627 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4c3f3eba-e695-11ee-95c9-87774f69a715 -- bash -c 'ceph orch ls' 2024-03-20T08:47:40.368 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/4c3f3eba-e695-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T08:47:40.582 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:47:40 smithi194 ceph-mon[27237]: from='client.14805 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-20T08:47:40.582 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:47:40 smithi194 ceph-mon[27237]: from='mgr.14637 ' entity='mgr.y' 2024-03-20T08:47:40.626 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:47:40 smithi183 ceph-mon[31860]: from='client.14805 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-20T08:47:40.626 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:47:40 smithi183 ceph-mon[31860]: from='mgr.14637 ' entity='mgr.y' 2024-03-20T08:47:40.626 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:47:40 smithi183 ceph-mon[25285]: from='client.14805 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-20T08:47:40.626 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:47:40 smithi183 ceph-mon[25285]: from='mgr.14637 ' entity='mgr.y' 2024-03-20T08:47:41.389 INFO:journalctl@ceph.iscsi.iscsi.a.smithi194.stdout:Mar 20 08:47:41 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-iscsi-iscsi-a[47467]: debug there is no tcmu-runner data available 2024-03-20T08:47:41.704 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:47:41 smithi183 ceph-mon[31860]: pgmap v40: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s; 22 B/s, 1 objects/s recovering 2024-03-20T08:47:41.704 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:47:41 smithi183 ceph-mon[31860]: from='mgr.14637 ' entity='mgr.y' 2024-03-20T08:47:41.705 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:47:41 smithi183 ceph-mon[25285]: pgmap v40: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s; 22 B/s, 1 objects/s recovering 2024-03-20T08:47:41.705 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:47:41 smithi183 ceph-mon[25285]: from='mgr.14637 ' entity='mgr.y' 2024-03-20T08:47:41.753 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:47:41 smithi194 ceph-mon[27237]: pgmap v40: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s; 22 B/s, 1 objects/s recovering 2024-03-20T08:47:41.754 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:47:41 smithi194 ceph-mon[27237]: from='mgr.14637 ' entity='mgr.y' 2024-03-20T08:47:42.523 INFO:teuthology.orchestra.run.smithi183.stdout:NAME PORTS RUNNING REFRESHED AGE PLACEMENT 2024-03-20T08:47:42.523 INFO:teuthology.orchestra.run.smithi183.stdout:agent 0/2 - 72s * 2024-03-20T08:47:42.523 INFO:teuthology.orchestra.run.smithi183.stdout:alertmanager ?:9093,9094 0/1 - 93s smithi183=a;count:1 2024-03-20T08:47:42.524 INFO:teuthology.orchestra.run.smithi183.stdout:grafana ?:3000 0/1 - 90s smithi194=a;count:1 2024-03-20T08:47:42.524 INFO:teuthology.orchestra.run.smithi183.stdout:iscsi.datapool ?:5000 1/1 95s ago 106s smithi194=iscsi.a;count:1 2024-03-20T08:47:42.524 INFO:teuthology.orchestra.run.smithi183.stdout:mgr 2/2 95s ago 5m smithi183=y;smithi194=x;count:2 2024-03-20T08:47:42.524 INFO:teuthology.orchestra.run.smithi183.stdout:mon 3/3 95s ago 6m smithi183:172.21.15.183=a;smithi183:[v2:172.21.15.183:3301,v1:172.21.15.183:6790]=c;smithi194:172.21.15.194=b;count:3 2024-03-20T08:47:42.524 INFO:teuthology.orchestra.run.smithi183.stdout:node-exporter ?:9100 0/2 - 97s smithi183=a;smithi194=b;count:2 2024-03-20T08:47:42.524 INFO:teuthology.orchestra.run.smithi183.stdout:osd 8 95s ago - 2024-03-20T08:47:42.524 INFO:teuthology.orchestra.run.smithi183.stdout:prometheus ?:9095 0/1 - 101s smithi194=a;count:1 2024-03-20T08:47:42.524 INFO:teuthology.orchestra.run.smithi183.stdout:rgw.foo.a ?:80 1/1 90s ago 114s smithi183=foo.a;count:1 2024-03-20T08:47:42.665 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:47:42 smithi183 ceph-mon[25285]: from='client.14559 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-20T08:47:42.665 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:47:42 smithi183 ceph-mon[31860]: from='client.14559 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-20T08:47:42.674 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:47:42 smithi194 ceph-mon[27237]: from='client.14559 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-20T08:47:43.510 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4c3f3eba-e695-11ee-95c9-87774f69a715 -- bash -c 'ceph orch host ls' 2024-03-20T08:47:43.565 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:47:43 smithi194 ceph-mon[27237]: pgmap v41: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 936 B/s rd, 0 op/s; 20 B/s, 1 objects/s recovering 2024-03-20T08:47:43.704 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:47:43 smithi183 ceph-mon[25285]: pgmap v41: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 936 B/s rd, 0 op/s; 20 B/s, 1 objects/s recovering 2024-03-20T08:47:43.705 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:47:43 smithi183 ceph-mon[31860]: pgmap v41: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 936 B/s rd, 0 op/s; 20 B/s, 1 objects/s recovering 2024-03-20T08:47:44.207 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/4c3f3eba-e695-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T08:47:44.704 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:47:44 smithi183 ceph-mon[25285]: from='client.14811 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-20T08:47:44.705 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:47:44 smithi183 ceph-mon[31860]: from='client.14811 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-20T08:47:44.888 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:47:44 smithi194 ceph-mon[27237]: from='client.14811 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-20T08:47:45.557 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:47:45 smithi194 ceph-mon[27237]: pgmap v42: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 18 B/s, 0 objects/s recovering 2024-03-20T08:47:45.557 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:47:45 smithi194 ceph-mon[27237]: from='mgr.14637 ' entity='mgr.y' 2024-03-20T08:47:45.557 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:47:45 smithi194 ceph-mon[27237]: from='mgr.14637 172.21.15.183:0/885080955' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-20T08:47:45.704 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:47:45 smithi183 ceph-mon[25285]: pgmap v42: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 18 B/s, 0 objects/s recovering 2024-03-20T08:47:45.705 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:47:45 smithi183 ceph-mon[25285]: from='mgr.14637 ' entity='mgr.y' 2024-03-20T08:47:45.705 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:47:45 smithi183 ceph-mon[25285]: from='mgr.14637 172.21.15.183:0/885080955' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-20T08:47:45.705 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:47:45 smithi183 ceph-mon[31860]: pgmap v42: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 18 B/s, 0 objects/s recovering 2024-03-20T08:47:45.705 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:47:45 smithi183 ceph-mon[31860]: from='mgr.14637 ' entity='mgr.y' 2024-03-20T08:47:45.705 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:47:45 smithi183 ceph-mon[31860]: from='mgr.14637 172.21.15.183:0/885080955' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-20T08:47:47.204 INFO:journalctl@ceph.mgr.y.smithi183.stdout:Mar 20 08:47:46 smithi183 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-mgr-y[25513]: ::ffff:172.21.15.194 - - [20/Mar/2024:08:47:46] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-20T08:47:47.445 INFO:teuthology.orchestra.run.smithi183.stdout:HOST ADDR LABELS STATUS 2024-03-20T08:47:47.446 INFO:teuthology.orchestra.run.smithi183.stdout:smithi183 172.21.15.183 2024-03-20T08:47:47.446 INFO:teuthology.orchestra.run.smithi183.stdout:smithi194 172.21.15.194 2024-03-20T08:47:47.446 INFO:teuthology.orchestra.run.smithi183.stdout:2 hosts in cluster 2024-03-20T08:47:47.639 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:47:47 smithi194 ceph-mon[27237]: pgmap v43: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 18 B/s, 0 objects/s recovering 2024-03-20T08:47:47.639 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:47:47 smithi194 ceph-mon[27237]: from='mgr.14637 ' entity='mgr.y' 2024-03-20T08:47:47.639 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:47:47 smithi194 ceph-mon[27237]: from='mgr.14637 ' entity='mgr.y' 2024-03-20T08:47:47.704 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:47:47 smithi183 ceph-mon[25285]: pgmap v43: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 18 B/s, 0 objects/s recovering 2024-03-20T08:47:47.704 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:47:47 smithi183 ceph-mon[25285]: from='mgr.14637 ' entity='mgr.y' 2024-03-20T08:47:47.704 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:47:47 smithi183 ceph-mon[25285]: from='mgr.14637 ' entity='mgr.y' 2024-03-20T08:47:47.705 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:47:47 smithi183 ceph-mon[31860]: pgmap v43: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 18 B/s, 0 objects/s recovering 2024-03-20T08:47:47.705 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:47:47 smithi183 ceph-mon[31860]: from='mgr.14637 ' entity='mgr.y' 2024-03-20T08:47:47.705 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:47:47 smithi183 ceph-mon[31860]: from='mgr.14637 ' entity='mgr.y' 2024-03-20T08:47:48.391 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4c3f3eba-e695-11ee-95c9-87774f69a715 -- bash -c 'ceph orch device ls' 2024-03-20T08:47:48.454 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:47:48 smithi183 ceph-mon[25285]: from='mgr.14637 ' entity='mgr.y' 2024-03-20T08:47:48.455 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:47:48 smithi183 ceph-mon[25285]: from='mgr.14637 ' entity='mgr.y' 2024-03-20T08:47:48.455 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:47:48 smithi183 ceph-mon[25285]: from='client.24604 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-20T08:47:48.455 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:47:48 smithi183 ceph-mon[31860]: from='mgr.14637 ' entity='mgr.y' 2024-03-20T08:47:48.455 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:47:48 smithi183 ceph-mon[31860]: from='mgr.14637 ' entity='mgr.y' 2024-03-20T08:47:48.455 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:47:48 smithi183 ceph-mon[31860]: from='client.24604 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-20T08:47:48.639 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:47:48 smithi194 ceph-mon[27237]: from='mgr.14637 ' entity='mgr.y' 2024-03-20T08:47:48.639 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:47:48 smithi194 ceph-mon[27237]: from='mgr.14637 ' entity='mgr.y' 2024-03-20T08:47:48.639 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:47:48 smithi194 ceph-mon[27237]: from='client.24604 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-20T08:47:49.389 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:47:49 smithi194 ceph-mon[27237]: pgmap v44: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-20T08:47:49.605 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/4c3f3eba-e695-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T08:47:49.704 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:47:49 smithi183 ceph-mon[25285]: pgmap v44: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-20T08:47:49.705 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:47:49 smithi183 ceph-mon[31860]: pgmap v44: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-20T08:47:51.329 INFO:journalctl@ceph.iscsi.iscsi.a.smithi194.stdout:Mar 20 08:47:51 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-iscsi-iscsi-a[47467]: debug there is no tcmu-runner data available 2024-03-20T08:47:51.454 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:47:51 smithi183 ceph-mon[31860]: pgmap v45: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-20T08:47:51.455 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:47:51 smithi183 ceph-mon[31860]: from='mgr.14637 ' entity='mgr.y' 2024-03-20T08:47:51.455 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:47:51 smithi183 ceph-mon[31860]: from='mgr.14637 ' entity='mgr.y' 2024-03-20T08:47:51.455 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:47:51 smithi183 ceph-mon[25285]: pgmap v45: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-20T08:47:51.455 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:47:51 smithi183 ceph-mon[25285]: from='mgr.14637 ' entity='mgr.y' 2024-03-20T08:47:51.455 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:47:51 smithi183 ceph-mon[25285]: from='mgr.14637 ' entity='mgr.y' 2024-03-20T08:47:51.638 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:47:51 smithi194 ceph-mon[27237]: pgmap v45: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-20T08:47:51.639 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:47:51 smithi194 ceph-mon[27237]: from='mgr.14637 ' entity='mgr.y' 2024-03-20T08:47:51.639 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:47:51 smithi194 ceph-mon[27237]: from='mgr.14637 ' entity='mgr.y' 2024-03-20T08:47:52.389 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:47:52 smithi194 ceph-mon[27237]: from='client.14559 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-20T08:47:52.454 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:47:52 smithi183 ceph-mon[25285]: from='client.14559 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-20T08:47:52.455 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:47:52 smithi183 ceph-mon[31860]: from='client.14559 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-20T08:47:52.882 INFO:teuthology.orchestra.run.smithi183.stdout:HOST PATH TYPE DEVICE ID SIZE AVAILABLE REFRESHED REJECT REASONS 2024-03-20T08:47:52.882 INFO:teuthology.orchestra.run.smithi183.stdout:smithi183 /dev/nvme0n1 ssd INTEL_SSDPEDMD400G4_PHFT620500QJ400BGN 372G No 2m ago Has a FileSystem, LVM detected 2024-03-20T08:47:52.882 INFO:teuthology.orchestra.run.smithi183.stdout:smithi183 /dev/nvme1n1 ssd Linux_c6a13abe82e79d34211d 89.4G No 2m ago Has a FileSystem, Insufficient space (<10 extents) on vgs, LVM detected 2024-03-20T08:47:52.882 INFO:teuthology.orchestra.run.smithi183.stdout:smithi183 /dev/nvme2n1 ssd Linux_2df36353822a1af07008 89.4G No 2m ago Has a FileSystem, Insufficient space (<10 extents) on vgs, LVM detected 2024-03-20T08:47:52.882 INFO:teuthology.orchestra.run.smithi183.stdout:smithi183 /dev/nvme3n1 ssd Linux_0558c7b77ddc7e9de393 89.4G No 2m ago Has a FileSystem, Insufficient space (<10 extents) on vgs, LVM detected 2024-03-20T08:47:52.882 INFO:teuthology.orchestra.run.smithi183.stdout:smithi183 /dev/nvme4n1 ssd Linux_0c043d4074e8718332b1 89.4G No 2m ago Has a FileSystem, Insufficient space (<10 extents) on vgs, LVM detected 2024-03-20T08:47:52.882 INFO:teuthology.orchestra.run.smithi183.stdout:smithi183 /dev/sda hdd ATA_ST1000NM0033-9ZM173_Z1W5XFFN 931G No 2m ago Has partitions 2024-03-20T08:47:52.883 INFO:teuthology.orchestra.run.smithi183.stdout:smithi194 /dev/nvme0n1 ssd INTEL_SSDPEDMD400G4_CVFT6240002A400BGN 372G No 105s ago Has a FileSystem, LVM detected 2024-03-20T08:47:52.883 INFO:teuthology.orchestra.run.smithi183.stdout:smithi194 /dev/nvme1n1 ssd Linux_9235e01bc4fe137c7840 89.4G No 105s ago Has a FileSystem, Insufficient space (<10 extents) on vgs, LVM detected 2024-03-20T08:47:52.883 INFO:teuthology.orchestra.run.smithi183.stdout:smithi194 /dev/nvme2n1 ssd Linux_71a165ac3a14b88708e4 89.4G No 105s ago Has a FileSystem, Insufficient space (<10 extents) on vgs, LVM detected 2024-03-20T08:47:52.883 INFO:teuthology.orchestra.run.smithi183.stdout:smithi194 /dev/nvme3n1 ssd Linux_009bba8792f40b5d0ea3 89.4G No 105s ago Has a FileSystem, Insufficient space (<10 extents) on vgs, LVM detected 2024-03-20T08:47:52.883 INFO:teuthology.orchestra.run.smithi183.stdout:smithi194 /dev/nvme4n1 ssd Linux_d8dd6ecf1af39479d860 89.4G No 105s ago Has a FileSystem, Insufficient space (<10 extents) on vgs, LVM detected 2024-03-20T08:47:52.883 INFO:teuthology.orchestra.run.smithi183.stdout:smithi194 /dev/sda hdd ATA_ST1000NM0033-9ZM173_Z1W5XNHZ 931G No 105s ago Has partitions 2024-03-20T08:47:53.454 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:47:53 smithi183 ceph-mon[31860]: pgmap v46: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-20T08:47:53.455 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:47:53 smithi183 ceph-mon[25285]: pgmap v46: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-20T08:47:53.602 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4c3f3eba-e695-11ee-95c9-87774f69a715 -- bash -c 'ceph orch ls --format yaml' 2024-03-20T08:47:53.639 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:47:53 smithi194 ceph-mon[27237]: pgmap v46: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-20T08:47:53.959 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/4c3f3eba-e695-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T08:47:54.639 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:47:54 smithi194 ceph-mon[27237]: from='client.14823 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-20T08:47:54.639 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:47:54 smithi194 ceph-mon[27237]: from='mgr.14637 ' entity='mgr.y' 2024-03-20T08:47:54.639 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:47:54 smithi194 ceph-mon[27237]: from='mgr.14637 ' entity='mgr.y' 2024-03-20T08:47:54.704 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:47:54 smithi183 ceph-mon[25285]: from='client.14823 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-20T08:47:54.704 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:47:54 smithi183 ceph-mon[25285]: from='mgr.14637 ' entity='mgr.y' 2024-03-20T08:47:54.705 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:47:54 smithi183 ceph-mon[25285]: from='mgr.14637 ' entity='mgr.y' 2024-03-20T08:47:54.705 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:47:54 smithi183 ceph-mon[31860]: from='client.14823 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-20T08:47:54.705 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:47:54 smithi183 ceph-mon[31860]: from='mgr.14637 ' entity='mgr.y' 2024-03-20T08:47:54.705 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:47:54 smithi183 ceph-mon[31860]: from='mgr.14637 ' entity='mgr.y' 2024-03-20T08:47:55.454 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:47:55 smithi183 ceph-mon[25285]: pgmap v47: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-20T08:47:55.454 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:47:55 smithi183 ceph-mon[31860]: pgmap v47: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-20T08:47:55.638 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:47:55 smithi194 ceph-mon[27237]: pgmap v47: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-20T08:47:56.454 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:47:56 smithi183 ceph-mon[25285]: Detected new or changed devices on smithi194 2024-03-20T08:47:56.454 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:47:56 smithi183 ceph-mon[25285]: from='mgr.14637 ' entity='mgr.y' 2024-03-20T08:47:56.454 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:47:56 smithi183 ceph-mon[25285]: from='mgr.14637 ' entity='mgr.y' 2024-03-20T08:47:56.455 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:47:56 smithi183 ceph-mon[31860]: Detected new or changed devices on smithi194 2024-03-20T08:47:56.455 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:47:56 smithi183 ceph-mon[31860]: from='mgr.14637 ' entity='mgr.y' 2024-03-20T08:47:56.455 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:47:56 smithi183 ceph-mon[31860]: from='mgr.14637 ' entity='mgr.y' 2024-03-20T08:47:56.526 INFO:teuthology.orchestra.run.smithi183.stdout:service_type: agent 2024-03-20T08:47:56.526 INFO:teuthology.orchestra.run.smithi183.stdout:service_name: agent 2024-03-20T08:47:56.526 INFO:teuthology.orchestra.run.smithi183.stdout:placement: 2024-03-20T08:47:56.526 INFO:teuthology.orchestra.run.smithi183.stdout: host_pattern: '*' 2024-03-20T08:47:56.526 INFO:teuthology.orchestra.run.smithi183.stdout:status: 2024-03-20T08:47:56.526 INFO:teuthology.orchestra.run.smithi183.stdout: created: '2024-03-20T08:46:29.656671Z' 2024-03-20T08:47:56.526 INFO:teuthology.orchestra.run.smithi183.stdout: last_refresh: '2024-03-20T08:47:46.263093Z' 2024-03-20T08:47:56.527 INFO:teuthology.orchestra.run.smithi183.stdout: running: 2 2024-03-20T08:47:56.527 INFO:teuthology.orchestra.run.smithi183.stdout: size: 2 2024-03-20T08:47:56.527 INFO:teuthology.orchestra.run.smithi183.stdout:events: 2024-03-20T08:47:56.527 INFO:teuthology.orchestra.run.smithi183.stdout:- 2024-03-20T08:46:30.958044Z service:agent [INFO] "service was created" 2024-03-20T08:47:56.527 INFO:teuthology.orchestra.run.smithi183.stdout:--- 2024-03-20T08:47:56.527 INFO:teuthology.orchestra.run.smithi183.stdout:service_type: alertmanager 2024-03-20T08:47:56.527 INFO:teuthology.orchestra.run.smithi183.stdout:service_name: alertmanager 2024-03-20T08:47:56.527 INFO:teuthology.orchestra.run.smithi183.stdout:placement: 2024-03-20T08:47:56.527 INFO:teuthology.orchestra.run.smithi183.stdout: count: 1 2024-03-20T08:47:56.527 INFO:teuthology.orchestra.run.smithi183.stdout: hosts: 2024-03-20T08:47:56.527 INFO:teuthology.orchestra.run.smithi183.stdout: - smithi183=a 2024-03-20T08:47:56.527 INFO:teuthology.orchestra.run.smithi183.stdout:status: 2024-03-20T08:47:56.527 INFO:teuthology.orchestra.run.smithi183.stdout: created: '2024-03-20T08:46:09.089230Z' 2024-03-20T08:47:56.527 INFO:teuthology.orchestra.run.smithi183.stdout: last_refresh: '2024-03-20T08:47:50.637752Z' 2024-03-20T08:47:56.527 INFO:teuthology.orchestra.run.smithi183.stdout: ports: 2024-03-20T08:47:56.527 INFO:teuthology.orchestra.run.smithi183.stdout: - 9093 2024-03-20T08:47:56.528 INFO:teuthology.orchestra.run.smithi183.stdout: - 9094 2024-03-20T08:47:56.528 INFO:teuthology.orchestra.run.smithi183.stdout: running: 1 2024-03-20T08:47:56.528 INFO:teuthology.orchestra.run.smithi183.stdout: size: 1 2024-03-20T08:47:56.528 INFO:teuthology.orchestra.run.smithi183.stdout:events: 2024-03-20T08:47:56.528 INFO:teuthology.orchestra.run.smithi183.stdout:- 2024-03-20T08:47:05.507481Z service:alertmanager [INFO] "service was created" 2024-03-20T08:47:56.528 INFO:teuthology.orchestra.run.smithi183.stdout:--- 2024-03-20T08:47:56.528 INFO:teuthology.orchestra.run.smithi183.stdout:service_type: grafana 2024-03-20T08:47:56.528 INFO:teuthology.orchestra.run.smithi183.stdout:service_name: grafana 2024-03-20T08:47:56.528 INFO:teuthology.orchestra.run.smithi183.stdout:placement: 2024-03-20T08:47:56.528 INFO:teuthology.orchestra.run.smithi183.stdout: count: 1 2024-03-20T08:47:56.528 INFO:teuthology.orchestra.run.smithi183.stdout: hosts: 2024-03-20T08:47:56.528 INFO:teuthology.orchestra.run.smithi183.stdout: - smithi194=a 2024-03-20T08:47:56.528 INFO:teuthology.orchestra.run.smithi183.stdout:spec: 2024-03-20T08:47:56.528 INFO:teuthology.orchestra.run.smithi183.stdout: anonymous_access: true 2024-03-20T08:47:56.528 INFO:teuthology.orchestra.run.smithi183.stdout: protocol: https 2024-03-20T08:47:56.528 INFO:teuthology.orchestra.run.smithi183.stdout:status: 2024-03-20T08:47:56.529 INFO:teuthology.orchestra.run.smithi183.stdout: created: '2024-03-20T08:46:12.503905Z' 2024-03-20T08:47:56.529 INFO:teuthology.orchestra.run.smithi183.stdout: last_refresh: '2024-03-20T08:47:46.263792Z' 2024-03-20T08:47:56.529 INFO:teuthology.orchestra.run.smithi183.stdout: ports: 2024-03-20T08:47:56.529 INFO:teuthology.orchestra.run.smithi183.stdout: - 3000 2024-03-20T08:47:56.529 INFO:teuthology.orchestra.run.smithi183.stdout: running: 1 2024-03-20T08:47:56.529 INFO:teuthology.orchestra.run.smithi183.stdout: size: 1 2024-03-20T08:47:56.529 INFO:teuthology.orchestra.run.smithi183.stdout:events: 2024-03-20T08:47:56.529 INFO:teuthology.orchestra.run.smithi183.stdout:- 2024-03-20T08:47:23.573186Z service:grafana [INFO] "service was created" 2024-03-20T08:47:56.529 INFO:teuthology.orchestra.run.smithi183.stdout:--- 2024-03-20T08:47:56.529 INFO:teuthology.orchestra.run.smithi183.stdout:service_type: iscsi 2024-03-20T08:47:56.529 INFO:teuthology.orchestra.run.smithi183.stdout:service_id: datapool 2024-03-20T08:47:56.529 INFO:teuthology.orchestra.run.smithi183.stdout:service_name: iscsi.datapool 2024-03-20T08:47:56.529 INFO:teuthology.orchestra.run.smithi183.stdout:placement: 2024-03-20T08:47:56.529 INFO:teuthology.orchestra.run.smithi183.stdout: count: 1 2024-03-20T08:47:56.529 INFO:teuthology.orchestra.run.smithi183.stdout: hosts: 2024-03-20T08:47:56.530 INFO:teuthology.orchestra.run.smithi183.stdout: - smithi194=iscsi.a 2024-03-20T08:47:56.530 INFO:teuthology.orchestra.run.smithi183.stdout:spec: 2024-03-20T08:47:56.530 INFO:teuthology.orchestra.run.smithi183.stdout: api_password: admin 2024-03-20T08:47:56.530 INFO:teuthology.orchestra.run.smithi183.stdout: api_port: 5000 2024-03-20T08:47:56.530 INFO:teuthology.orchestra.run.smithi183.stdout: api_user: admin 2024-03-20T08:47:56.530 INFO:teuthology.orchestra.run.smithi183.stdout: pool: datapool 2024-03-20T08:47:56.530 INFO:teuthology.orchestra.run.smithi183.stdout: trusted_ip_list: 172.21.15.194 2024-03-20T08:47:56.530 INFO:teuthology.orchestra.run.smithi183.stdout:status: 2024-03-20T08:47:56.530 INFO:teuthology.orchestra.run.smithi183.stdout: created: '2024-03-20T08:45:56.472146Z' 2024-03-20T08:47:56.530 INFO:teuthology.orchestra.run.smithi183.stdout: last_refresh: '2024-03-20T08:47:46.263653Z' 2024-03-20T08:47:56.530 INFO:teuthology.orchestra.run.smithi183.stdout: ports: 2024-03-20T08:47:56.530 INFO:teuthology.orchestra.run.smithi183.stdout: - 5000 2024-03-20T08:47:56.530 INFO:teuthology.orchestra.run.smithi183.stdout: running: 1 2024-03-20T08:47:56.530 INFO:teuthology.orchestra.run.smithi183.stdout: size: 1 2024-03-20T08:47:56.530 INFO:teuthology.orchestra.run.smithi183.stdout:--- 2024-03-20T08:47:56.531 INFO:teuthology.orchestra.run.smithi183.stdout:service_type: mgr 2024-03-20T08:47:56.531 INFO:teuthology.orchestra.run.smithi183.stdout:service_name: mgr 2024-03-20T08:47:56.531 INFO:teuthology.orchestra.run.smithi183.stdout:placement: 2024-03-20T08:47:56.531 INFO:teuthology.orchestra.run.smithi183.stdout: count: 2 2024-03-20T08:47:56.531 INFO:teuthology.orchestra.run.smithi183.stdout: hosts: 2024-03-20T08:47:56.531 INFO:teuthology.orchestra.run.smithi183.stdout: - smithi183=y 2024-03-20T08:47:56.531 INFO:teuthology.orchestra.run.smithi183.stdout: - smithi194=x 2024-03-20T08:47:56.531 INFO:teuthology.orchestra.run.smithi183.stdout:status: 2024-03-20T08:47:56.531 INFO:teuthology.orchestra.run.smithi183.stdout: created: '2024-03-20T08:42:00.480331Z' 2024-03-20T08:47:56.531 INFO:teuthology.orchestra.run.smithi183.stdout: last_refresh: '2024-03-20T08:47:46.263280Z' 2024-03-20T08:47:56.531 INFO:teuthology.orchestra.run.smithi183.stdout: running: 2 2024-03-20T08:47:56.531 INFO:teuthology.orchestra.run.smithi183.stdout: size: 2 2024-03-20T08:47:56.531 INFO:teuthology.orchestra.run.smithi183.stdout:--- 2024-03-20T08:47:56.531 INFO:teuthology.orchestra.run.smithi183.stdout:service_type: mon 2024-03-20T08:47:56.531 INFO:teuthology.orchestra.run.smithi183.stdout:service_name: mon 2024-03-20T08:47:56.531 INFO:teuthology.orchestra.run.smithi183.stdout:placement: 2024-03-20T08:47:56.532 INFO:teuthology.orchestra.run.smithi183.stdout: count: 3 2024-03-20T08:47:56.532 INFO:teuthology.orchestra.run.smithi183.stdout: hosts: 2024-03-20T08:47:56.532 INFO:teuthology.orchestra.run.smithi183.stdout: - smithi183:172.21.15.183=a 2024-03-20T08:47:56.532 INFO:teuthology.orchestra.run.smithi183.stdout: - smithi183:[v2:172.21.15.183:3301,v1:172.21.15.183:6790]=c 2024-03-20T08:47:56.532 INFO:teuthology.orchestra.run.smithi183.stdout: - smithi194:172.21.15.194=b 2024-03-20T08:47:56.532 INFO:teuthology.orchestra.run.smithi183.stdout:status: 2024-03-20T08:47:56.532 INFO:teuthology.orchestra.run.smithi183.stdout: created: '2024-03-20T08:41:31.559874Z' 2024-03-20T08:47:56.532 INFO:teuthology.orchestra.run.smithi183.stdout: last_refresh: '2024-03-20T08:47:46.263199Z' 2024-03-20T08:47:56.532 INFO:teuthology.orchestra.run.smithi183.stdout: running: 3 2024-03-20T08:47:56.532 INFO:teuthology.orchestra.run.smithi183.stdout: size: 3 2024-03-20T08:47:56.532 INFO:teuthology.orchestra.run.smithi183.stdout:--- 2024-03-20T08:47:56.532 INFO:teuthology.orchestra.run.smithi183.stdout:service_type: node-exporter 2024-03-20T08:47:56.532 INFO:teuthology.orchestra.run.smithi183.stdout:service_name: node-exporter 2024-03-20T08:47:56.532 INFO:teuthology.orchestra.run.smithi183.stdout:placement: 2024-03-20T08:47:56.532 INFO:teuthology.orchestra.run.smithi183.stdout: count: 2 2024-03-20T08:47:56.532 INFO:teuthology.orchestra.run.smithi183.stdout: hosts: 2024-03-20T08:47:56.533 INFO:teuthology.orchestra.run.smithi183.stdout: - smithi183=a 2024-03-20T08:47:56.533 INFO:teuthology.orchestra.run.smithi183.stdout: - smithi194=b 2024-03-20T08:47:56.533 INFO:teuthology.orchestra.run.smithi183.stdout:status: 2024-03-20T08:47:56.533 INFO:teuthology.orchestra.run.smithi183.stdout: created: '2024-03-20T08:46:05.504447Z' 2024-03-20T08:47:56.533 INFO:teuthology.orchestra.run.smithi183.stdout: last_refresh: '2024-03-20T08:47:46.263862Z' 2024-03-20T08:47:56.533 INFO:teuthology.orchestra.run.smithi183.stdout: ports: 2024-03-20T08:47:56.533 INFO:teuthology.orchestra.run.smithi183.stdout: - 9100 2024-03-20T08:47:56.533 INFO:teuthology.orchestra.run.smithi183.stdout: running: 2 2024-03-20T08:47:56.533 INFO:teuthology.orchestra.run.smithi183.stdout: size: 2 2024-03-20T08:47:56.533 INFO:teuthology.orchestra.run.smithi183.stdout:events: 2024-03-20T08:47:56.533 INFO:teuthology.orchestra.run.smithi183.stdout:- 2024-03-20T08:47:35.401804Z service:node-exporter [INFO] "service was created" 2024-03-20T08:47:56.533 INFO:teuthology.orchestra.run.smithi183.stdout:--- 2024-03-20T08:47:56.533 INFO:teuthology.orchestra.run.smithi183.stdout:service_type: osd 2024-03-20T08:47:56.533 INFO:teuthology.orchestra.run.smithi183.stdout:service_name: osd 2024-03-20T08:47:56.533 INFO:teuthology.orchestra.run.smithi183.stdout:unmanaged: true 2024-03-20T08:47:56.534 INFO:teuthology.orchestra.run.smithi183.stdout:spec: 2024-03-20T08:47:56.534 INFO:teuthology.orchestra.run.smithi183.stdout: filter_logic: AND 2024-03-20T08:47:56.534 INFO:teuthology.orchestra.run.smithi183.stdout: objectstore: bluestore 2024-03-20T08:47:56.534 INFO:teuthology.orchestra.run.smithi183.stdout:status: 2024-03-20T08:47:56.534 INFO:teuthology.orchestra.run.smithi183.stdout: container_image_id: 9ceba81fb2c8d8682cb159527017d591637b153b3aa4a390ffa8586db6dcda00 2024-03-20T08:47:56.534 INFO:teuthology.orchestra.run.smithi183.stdout: container_image_name: quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:976a8b2d06e3c6bfd12488297e8584dd704cc06c3e976c7e0f718c13e238e912 2024-03-20T08:47:56.534 INFO:teuthology.orchestra.run.smithi183.stdout: last_refresh: '2024-03-20T08:47:46.263370Z' 2024-03-20T08:47:56.534 INFO:teuthology.orchestra.run.smithi183.stdout: running: 8 2024-03-20T08:47:56.534 INFO:teuthology.orchestra.run.smithi183.stdout: size: 8 2024-03-20T08:47:56.534 INFO:teuthology.orchestra.run.smithi183.stdout:--- 2024-03-20T08:47:56.534 INFO:teuthology.orchestra.run.smithi183.stdout:service_type: prometheus 2024-03-20T08:47:56.534 INFO:teuthology.orchestra.run.smithi183.stdout:service_name: prometheus 2024-03-20T08:47:56.534 INFO:teuthology.orchestra.run.smithi183.stdout:placement: 2024-03-20T08:47:56.534 INFO:teuthology.orchestra.run.smithi183.stdout: count: 1 2024-03-20T08:47:56.534 INFO:teuthology.orchestra.run.smithi183.stdout: hosts: 2024-03-20T08:47:56.534 INFO:teuthology.orchestra.run.smithi183.stdout: - smithi194=a 2024-03-20T08:47:56.535 INFO:teuthology.orchestra.run.smithi183.stdout:status: 2024-03-20T08:47:56.535 INFO:teuthology.orchestra.run.smithi183.stdout: created: '2024-03-20T08:46:00.534562Z' 2024-03-20T08:47:56.535 INFO:teuthology.orchestra.run.smithi183.stdout: last_refresh: '2024-03-20T08:47:46.263722Z' 2024-03-20T08:47:56.535 INFO:teuthology.orchestra.run.smithi183.stdout: ports: 2024-03-20T08:47:56.535 INFO:teuthology.orchestra.run.smithi183.stdout: - 9095 2024-03-20T08:47:56.535 INFO:teuthology.orchestra.run.smithi183.stdout: running: 1 2024-03-20T08:47:56.535 INFO:teuthology.orchestra.run.smithi183.stdout: size: 1 2024-03-20T08:47:56.535 INFO:teuthology.orchestra.run.smithi183.stdout:events: 2024-03-20T08:47:56.535 INFO:teuthology.orchestra.run.smithi183.stdout:- 2024-03-20T08:47:35.410018Z service:prometheus [INFO] "service was created" 2024-03-20T08:47:56.535 INFO:teuthology.orchestra.run.smithi183.stdout:--- 2024-03-20T08:47:56.535 INFO:teuthology.orchestra.run.smithi183.stdout:service_type: rgw 2024-03-20T08:47:56.535 INFO:teuthology.orchestra.run.smithi183.stdout:service_id: foo.a 2024-03-20T08:47:56.535 INFO:teuthology.orchestra.run.smithi183.stdout:service_name: rgw.foo.a 2024-03-20T08:47:56.535 INFO:teuthology.orchestra.run.smithi183.stdout:placement: 2024-03-20T08:47:56.535 INFO:teuthology.orchestra.run.smithi183.stdout: count: 1 2024-03-20T08:47:56.535 INFO:teuthology.orchestra.run.smithi183.stdout: hosts: 2024-03-20T08:47:56.536 INFO:teuthology.orchestra.run.smithi183.stdout: - smithi183=foo.a 2024-03-20T08:47:56.536 INFO:teuthology.orchestra.run.smithi183.stdout:status: 2024-03-20T08:47:56.536 INFO:teuthology.orchestra.run.smithi183.stdout: created: '2024-03-20T08:45:48.233366Z' 2024-03-20T08:47:56.536 INFO:teuthology.orchestra.run.smithi183.stdout: last_refresh: '2024-03-20T08:47:50.637675Z' 2024-03-20T08:47:56.536 INFO:teuthology.orchestra.run.smithi183.stdout: ports: 2024-03-20T08:47:56.536 INFO:teuthology.orchestra.run.smithi183.stdout: - 80 2024-03-20T08:47:56.536 INFO:teuthology.orchestra.run.smithi183.stdout: running: 1 2024-03-20T08:47:56.536 INFO:teuthology.orchestra.run.smithi183.stdout: size: 1 2024-03-20T08:47:56.888 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:47:56 smithi194 ceph-mon[27237]: Detected new or changed devices on smithi194 2024-03-20T08:47:56.889 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:47:56 smithi194 ceph-mon[27237]: from='mgr.14637 ' entity='mgr.y' 2024-03-20T08:47:56.889 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:47:56 smithi194 ceph-mon[27237]: from='mgr.14637 ' entity='mgr.y' 2024-03-20T08:47:57.192 INFO:journalctl@ceph.mgr.y.smithi183.stdout:Mar 20 08:47:56 smithi183 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-mgr-y[25513]: ::ffff:172.21.15.194 - - [20/Mar/2024:08:47:56] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-20T08:47:57.454 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:47:57 smithi183 ceph-mon[25285]: pgmap v48: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-20T08:47:57.454 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:47:57 smithi183 ceph-mon[25285]: from='client.24722 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "yaml"}]: dispatch 2024-03-20T08:47:57.455 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:47:57 smithi183 ceph-mon[31860]: pgmap v48: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-20T08:47:57.455 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:47:57 smithi183 ceph-mon[31860]: from='client.24722 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "yaml"}]: dispatch 2024-03-20T08:47:57.788 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4c3f3eba-e695-11ee-95c9-87774f69a715 -- bash -c 'ceph orch ls | grep '"'"'^osd '"'"'' 2024-03-20T08:47:57.875 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:47:57 smithi194 ceph-mon[27237]: pgmap v48: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-20T08:47:57.875 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:47:57 smithi194 ceph-mon[27237]: from='client.24722 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "yaml"}]: dispatch 2024-03-20T08:47:58.084 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/4c3f3eba-e695-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T08:47:59.454 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:47:59 smithi183 ceph-mon[25285]: pgmap v49: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-20T08:47:59.455 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:47:59 smithi183 ceph-mon[31860]: pgmap v49: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-20T08:47:59.639 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:47:59 smithi194 ceph-mon[27237]: pgmap v49: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-20T08:48:00.198 INFO:teuthology.orchestra.run.smithi183.stdout:osd 8 13s ago - 2024-03-20T08:48:00.704 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:48:00 smithi183 ceph-mon[25285]: from='mgr.14637 172.21.15.183:0/885080955' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-20T08:48:00.704 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:48:00 smithi183 ceph-mon[31860]: from='mgr.14637 172.21.15.183:0/885080955' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-20T08:48:00.888 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:48:00 smithi194 ceph-mon[27237]: from='mgr.14637 172.21.15.183:0/885080955' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-20T08:48:01.268 DEBUG:teuthology.run_tasks:Unwinding manager cephadm 2024-03-20T08:48:01.278 INFO:tasks.cephadm:Teardown begin 2024-03-20T08:48:01.278 DEBUG:teuthology.orchestra.run.smithi183:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-03-20T08:48:01.310 DEBUG:teuthology.orchestra.run.smithi194:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-03-20T08:48:01.341 INFO:journalctl@ceph.iscsi.iscsi.a.smithi194.stdout:Mar 20 08:48:01 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-iscsi-iscsi-a[47467]: debug there is no tcmu-runner data available 2024-03-20T08:48:01.346 INFO:tasks.cephadm:Cleaning up testdir ceph.* files... 2024-03-20T08:48:01.347 DEBUG:teuthology.orchestra.run.smithi183:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-03-20T08:48:01.365 DEBUG:teuthology.orchestra.run.smithi194:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-03-20T08:48:01.403 INFO:tasks.cephadm:Stopping all daemons... 2024-03-20T08:48:01.404 INFO:tasks.cephadm.mon.a:Stopping mon.a... 2024-03-20T08:48:01.404 DEBUG:teuthology.orchestra.run.smithi183:> sudo systemctl stop ceph-4c3f3eba-e695-11ee-95c9-87774f69a715@mon.a 2024-03-20T08:48:01.551 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:48:01 smithi183 ceph-mon[25285]: pgmap v50: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-20T08:48:01.551 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:48:01 smithi183 ceph-mon[25285]: from='client.14835 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-20T08:48:01.552 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:48:01 smithi183 ceph-mon[25285]: from='mgr.14637 ' entity='mgr.y' 2024-03-20T08:48:01.552 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:48:01 smithi183 ceph-mon[25285]: from='mgr.14637 ' entity='mgr.y' 2024-03-20T08:48:01.552 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:48:01 smithi183 ceph-mon[25285]: from='mgr.14637 172.21.15.183:0/885080955' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-20T08:48:01.552 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:48:01 smithi183 ceph-mon[25285]: from='mgr.14637 172.21.15.183:0/885080955' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-20T08:48:01.552 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:48:01 smithi183 ceph-mon[25285]: from='mgr.14637 ' entity='mgr.y' 2024-03-20T08:48:01.552 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:48:01 smithi183 systemd[1]: Stopping Ceph mon.a for 4c3f3eba-e695-11ee-95c9-87774f69a715... 2024-03-20T08:48:01.552 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:48:01 smithi183 ceph-mon[31860]: pgmap v50: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-20T08:48:01.552 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:48:01 smithi183 ceph-mon[31860]: from='client.14835 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-20T08:48:01.552 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:48:01 smithi183 ceph-mon[31860]: from='mgr.14637 ' entity='mgr.y' 2024-03-20T08:48:01.552 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:48:01 smithi183 ceph-mon[31860]: from='mgr.14637 ' entity='mgr.y' 2024-03-20T08:48:01.552 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:48:01 smithi183 ceph-mon[31860]: from='mgr.14637 172.21.15.183:0/885080955' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-20T08:48:01.552 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:48:01 smithi183 ceph-mon[31860]: from='mgr.14637 172.21.15.183:0/885080955' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-20T08:48:01.552 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:48:01 smithi183 ceph-mon[31860]: from='mgr.14637 ' entity='mgr.y' 2024-03-20T08:48:01.639 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:48:01 smithi194 ceph-mon[27237]: pgmap v50: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-20T08:48:01.639 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:48:01 smithi194 ceph-mon[27237]: from='client.14835 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-20T08:48:01.639 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:48:01 smithi194 ceph-mon[27237]: from='mgr.14637 ' entity='mgr.y' 2024-03-20T08:48:01.639 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:48:01 smithi194 ceph-mon[27237]: from='mgr.14637 ' entity='mgr.y' 2024-03-20T08:48:01.639 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:48:01 smithi194 ceph-mon[27237]: from='mgr.14637 172.21.15.183:0/885080955' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-20T08:48:01.639 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:48:01 smithi194 ceph-mon[27237]: from='mgr.14637 172.21.15.183:0/885080955' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-20T08:48:01.639 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:48:01 smithi194 ceph-mon[27237]: from='mgr.14637 ' entity='mgr.y' 2024-03-20T08:48:02.517 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:48:02 smithi183 ceph-mon[25285]: Detected new or changed devices on smithi183 2024-03-20T08:48:02.517 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:48:02 smithi183 ceph-mon[25285]: pgmap v51: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 932 B/s rd, 0 op/s 2024-03-20T08:48:02.517 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:48:02 smithi183 ceph-mon[25285]: Reconfiguring alertmanager.a (dependencies changed)... 2024-03-20T08:48:02.517 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:48:02 smithi183 ceph-mon[25285]: Reconfiguring daemon alertmanager.a on smithi183 2024-03-20T08:48:02.518 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:48:02 smithi183 ceph-mon[25285]: from='client.14559 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-20T08:48:02.518 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:48:02 smithi183 ceph-mon[25285]: Health check failed: 2 Cephadm Agent(s) are not reporting. Hosts may be offline (CEPHADM_AGENT_DOWN) 2024-03-20T08:48:02.518 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:48:02 smithi183 ceph-mon[31860]: Detected new or changed devices on smithi183 2024-03-20T08:48:02.518 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:48:02 smithi183 ceph-mon[31860]: pgmap v51: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 932 B/s rd, 0 op/s 2024-03-20T08:48:02.518 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:48:02 smithi183 ceph-mon[31860]: Reconfiguring alertmanager.a (dependencies changed)... 2024-03-20T08:48:02.518 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:48:02 smithi183 ceph-mon[31860]: Reconfiguring daemon alertmanager.a on smithi183 2024-03-20T08:48:02.518 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:48:02 smithi183 ceph-mon[31860]: from='client.14559 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-20T08:48:02.518 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:48:02 smithi183 ceph-mon[31860]: Health check failed: 2 Cephadm Agent(s) are not reporting. Hosts may be offline (CEPHADM_AGENT_DOWN) 2024-03-20T08:48:02.888 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:48:02 smithi194 ceph-mon[27237]: Detected new or changed devices on smithi183 2024-03-20T08:48:02.889 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:48:02 smithi194 ceph-mon[27237]: pgmap v51: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 932 B/s rd, 0 op/s 2024-03-20T08:48:02.889 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:48:02 smithi194 ceph-mon[27237]: Reconfiguring alertmanager.a (dependencies changed)... 2024-03-20T08:48:02.889 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:48:02 smithi194 ceph-mon[27237]: Reconfiguring daemon alertmanager.a on smithi183 2024-03-20T08:48:02.889 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:48:02 smithi194 ceph-mon[27237]: from='client.14559 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-20T08:48:02.889 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:48:02 smithi194 ceph-mon[27237]: Health check failed: 2 Cephadm Agent(s) are not reporting. Hosts may be offline (CEPHADM_AGENT_DOWN) 2024-03-20T08:48:04.704 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:48:04 smithi183 ceph-mon[25285]: pgmap v52: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 932 B/s rd, 0 op/s 2024-03-20T08:48:04.705 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:48:04 smithi183 ceph-mon[31860]: pgmap v52: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 932 B/s rd, 0 op/s 2024-03-20T08:48:04.888 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:48:04 smithi194 ceph-mon[27237]: pgmap v52: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 932 B/s rd, 0 op/s 2024-03-20T08:48:05.068 INFO:journalctl@ceph.alertmanager.a.smithi183.stdout:Mar 20 08:48:04 smithi183 systemd[1]: Stopping Ceph alertmanager.a for 4c3f3eba-e695-11ee-95c9-87774f69a715... 2024-03-20T08:48:05.454 INFO:journalctl@ceph.alertmanager.a.smithi183.stdout:Mar 20 08:48:05 smithi183 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-alertmanager-a[60200]: ts=2024-03-20T08:48:05.262Z caller=main.go:583 level=info msg="Received SIGTERM, exiting gracefully..." 2024-03-20T08:48:05.454 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:48:05 smithi183 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-mon-a[25281]: 2024-03-20T08:48:05.067+0000 7fc03f4cd700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-03-20T08:48:05.455 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:48:05 smithi183 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-mon-a[25281]: 2024-03-20T08:48:05.067+0000 7fc03f4cd700 -1 mon.a@0(leader) e3 *** Got Signal Terminated *** 2024-03-20T08:48:05.455 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:48:05 smithi183 podman[71419]: 2024-03-20 08:48:05.282295769 +0000 UTC m=+1.655479570 container died 8c5d9ae3ddbe9fc75c7c8dd9549797b4bd09ef885926e3aacbd48b29041f5ee2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae, name=ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-mon-a, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, RELEASE=wip-yuri5-testing-2024-03-18-1144-e7e80a6, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_CLEAN=True, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, org.label-schema.build-date=20240307, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-20T08:48:05.954 INFO:journalctl@ceph.alertmanager.a.smithi183.stdout:Mar 20 08:48:05 smithi183 podman[71576]: 2024-03-20 08:48:05.712012904 +0000 UTC m=+0.605301758 container died 4915bdfc21cf8b2e25383805fbf6ac9bd8b63791095d9034e9f4bdc7d2a913bd (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-03-20T08:48:05.955 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:48:05 smithi183 podman[71419]: 2024-03-20 08:48:05.654179466 +0000 UTC m=+2.027363266 container cleanup 8c5d9ae3ddbe9fc75c7c8dd9549797b4bd09ef885926e3aacbd48b29041f5ee2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae, name=ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-mon-a, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_CLEAN=True, CEPH_POINT_RELEASE=, ceph=True, RELEASE=wip-yuri5-testing-2024-03-18-1144-e7e80a6, org.label-schema.license=GPLv2) 2024-03-20T08:48:05.955 INFO:journalctl@ceph.mon.a.smithi183.stdout:Mar 20 08:48:05 smithi183 bash[71419]: ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-mon-a 2024-03-20T08:48:06.415 INFO:journalctl@ceph.alertmanager.a.smithi183.stdout:Mar 20 08:48:06 smithi183 podman[71576]: 2024-03-20 08:48:06.147852101 +0000 UTC m=+1.041140998 container cleanup 4915bdfc21cf8b2e25383805fbf6ac9bd8b63791095d9034e9f4bdc7d2a913bd (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-03-20T08:48:06.415 INFO:journalctl@ceph.alertmanager.a.smithi183.stdout:Mar 20 08:48:06 smithi183 bash[71576]: ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-alertmanager-a 2024-03-20T08:48:06.451 DEBUG:teuthology.orchestra.run.smithi183:> sudo pkill -f 'journalctl -f -n 0 -u ceph-4c3f3eba-e695-11ee-95c9-87774f69a715@mon.a.service' 2024-03-20T08:48:06.521 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-20T08:48:06.521 INFO:tasks.cephadm.mon.a:Stopped mon.a 2024-03-20T08:48:06.522 INFO:tasks.cephadm.mon.b:Stopping mon.c... 2024-03-20T08:48:06.522 DEBUG:teuthology.orchestra.run.smithi183:> sudo systemctl stop ceph-4c3f3eba-e695-11ee-95c9-87774f69a715@mon.c 2024-03-20T08:48:06.683 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:48:06 smithi183 systemd[1]: Stopping Ceph mon.c for 4c3f3eba-e695-11ee-95c9-87774f69a715... 2024-03-20T08:48:06.939 INFO:journalctl@ceph.alertmanager.a.smithi183.stdout:Mar 20 08:48:06 smithi183 podman[71591]: 2024-03-20 08:48:06.683918829 +0000 UTC m=+1.396599992 container remove 4915bdfc21cf8b2e25383805fbf6ac9bd8b63791095d9034e9f4bdc7d2a913bd (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-03-20T08:48:06.939 INFO:journalctl@ceph.alertmanager.a.smithi183.stdout:Mar 20 08:48:06 smithi183 podman[71591]: 2024-03-20 08:48:06.793258701 +0000 UTC m=+1.505939875 volume remove 0ffb275b50edf3cf646fb980a3be04abceea11b366f62e92d65d69a4a4c4766e 2024-03-20T08:48:06.939 INFO:journalctl@ceph.alertmanager.a.smithi183.stdout:Mar 20 08:48:06 smithi183 systemd[1]: ceph-4c3f3eba-e695-11ee-95c9-87774f69a715@alertmanager.a.service: Deactivated successfully. 2024-03-20T08:48:06.939 INFO:journalctl@ceph.alertmanager.a.smithi183.stdout:Mar 20 08:48:06 smithi183 systemd[1]: Stopped Ceph alertmanager.a for 4c3f3eba-e695-11ee-95c9-87774f69a715. 2024-03-20T08:48:06.939 INFO:journalctl@ceph.alertmanager.a.smithi183.stdout:Mar 20 08:48:06 smithi183 systemd[1]: Starting Ceph alertmanager.a for 4c3f3eba-e695-11ee-95c9-87774f69a715... 2024-03-20T08:48:07.204 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:48:07 smithi183 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-mon-c[31856]: 2024-03-20T08:48:07.053+0000 7f7ceb21e700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.c -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-03-20T08:48:07.204 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:48:07 smithi183 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-mon-c[31856]: 2024-03-20T08:48:07.053+0000 7f7ceb21e700 -1 mon.c@2(peon) e3 *** Got Signal Terminated *** 2024-03-20T08:48:07.204 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:48:07 smithi183 podman[71766]: 2024-03-20 08:48:07.085086527 +0000 UTC m=+0.135450234 container died c5f899930acf7b1491b16ce5109556156eee62aa6429bc68a88d5b004157b894 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:976a8b2d06e3c6bfd12488297e8584dd704cc06c3e976c7e0f718c13e238e912, name=ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-mon-c, RELEASE=wip-yuri5-testing-2024-03-18-1144-e7e80a6, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.build-date=20240307, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-20T08:48:07.205 INFO:journalctl@ceph.mgr.y.smithi183.stdout:Mar 20 08:48:06 smithi183 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-mgr-y[25513]: ::ffff:172.21.15.194 - - [20/Mar/2024:08:48:06] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-20T08:48:07.205 INFO:journalctl@ceph.mgr.y.smithi183.stdout:Mar 20 08:48:07 smithi183 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-mgr-y[25513]: [20/Mar/2024:08:48:07] ENGINE Bus STOPPING 2024-03-20T08:48:07.205 INFO:journalctl@ceph.mgr.y.smithi183.stdout:Mar 20 08:48:07 smithi183 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-mgr-y[25513]: [20/Mar/2024:08:48:07] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-03-20T08:48:07.205 INFO:journalctl@ceph.mgr.y.smithi183.stdout:Mar 20 08:48:07 smithi183 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-mgr-y[25513]: [20/Mar/2024:08:48:07] ENGINE Bus STOPPED 2024-03-20T08:48:07.205 INFO:journalctl@ceph.mgr.y.smithi183.stdout:Mar 20 08:48:07 smithi183 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-mgr-y[25513]: [20/Mar/2024:08:48:07] ENGINE Bus STARTING 2024-03-20T08:48:07.205 INFO:journalctl@ceph.mgr.y.smithi183.stdout:Mar 20 08:48:07 smithi183 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-mgr-y[25513]: [20/Mar/2024:08:48:07] ENGINE Serving on http://:::9283 2024-03-20T08:48:07.205 INFO:journalctl@ceph.mgr.y.smithi183.stdout:Mar 20 08:48:07 smithi183 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-mgr-y[25513]: [20/Mar/2024:08:48:07] ENGINE Bus STARTED 2024-03-20T08:48:07.954 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:48:07 smithi183 podman[71766]: 2024-03-20 08:48:07.612895639 +0000 UTC m=+0.663259386 container cleanup c5f899930acf7b1491b16ce5109556156eee62aa6429bc68a88d5b004157b894 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:976a8b2d06e3c6bfd12488297e8584dd704cc06c3e976c7e0f718c13e238e912, name=ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-mon-c, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, RELEASE=wip-yuri5-testing-2024-03-18-1144-e7e80a6, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, ceph=True, org.label-schema.build-date=20240307, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0) 2024-03-20T08:48:07.954 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:48:07 smithi183 bash[71766]: ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-mon-c 2024-03-20T08:48:07.954 INFO:journalctl@ceph.mon.c.smithi183.stdout:Mar 20 08:48:07 smithi183 podman[71809]: 2024-03-20 08:48:07.839208355 +0000 UTC m=+0.757476947 container remove c5f899930acf7b1491b16ce5109556156eee62aa6429bc68a88d5b004157b894 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:976a8b2d06e3c6bfd12488297e8584dd704cc06c3e976c7e0f718c13e238e912, name=ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-mon-c, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_CLEAN=True, ceph=True, RELEASE=wip-yuri5-testing-2024-03-18-1144-e7e80a6, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240307, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-20T08:48:08.265 INFO:journalctl@ceph.alertmanager.a.smithi183.stdout:Mar 20 08:48:07 smithi183 podman[71841]: 2024-03-20 08:48:07.981663323 +0000 UTC m=+0.344265782 volume create 7c9d4b409337d1a7b07b7ae6c0c0c1b109abae5d7589001e0603cfd35b01e746 2024-03-20T08:48:08.265 INFO:journalctl@ceph.alertmanager.a.smithi183.stdout:Mar 20 08:48:07 smithi183 podman[71841]: 2024-03-20 08:48:07.890853923 +0000 UTC m=+0.253456402 image pull c8568f914cd25b2062c44e9f79f9c18da6e3b85fe0c47a12a2191c61426c2b19 quay.io/prometheus/alertmanager:v0.25.0 2024-03-20T08:48:08.265 INFO:journalctl@ceph.alertmanager.a.smithi183.stdout:Mar 20 08:48:08 smithi183 podman[71841]: 2024-03-20 08:48:08.082078837 +0000 UTC m=+0.444681342 container create 3aa83d988989e0cf6b2a0d2389fcf7b0e8343aad495911a18e5ca7cedfafe732 (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-03-20T08:48:08.682 INFO:journalctl@ceph.alertmanager.a.smithi183.stdout:Mar 20 08:48:08 smithi183 podman[71841]: 2024-03-20 08:48:08.384495001 +0000 UTC m=+0.747097461 container init 3aa83d988989e0cf6b2a0d2389fcf7b0e8343aad495911a18e5ca7cedfafe732 (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-03-20T08:48:08.682 INFO:journalctl@ceph.alertmanager.a.smithi183.stdout:Mar 20 08:48:08 smithi183 podman[71841]: 2024-03-20 08:48:08.387347525 +0000 UTC m=+0.749949995 container start 3aa83d988989e0cf6b2a0d2389fcf7b0e8343aad495911a18e5ca7cedfafe732 (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-03-20T08:48:08.682 INFO:journalctl@ceph.alertmanager.a.smithi183.stdout:Mar 20 08:48:08 smithi183 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-alertmanager-a[71868]: ts=2024-03-20T08:48:08.399Z caller=main.go:240 level=info msg="Starting Alertmanager" version="(version=0.25.0, branch=HEAD, revision=258fab7cdd551f2cf251ed0348f0ad7289aee789)" 2024-03-20T08:48:08.682 INFO:journalctl@ceph.alertmanager.a.smithi183.stdout:Mar 20 08:48:08 smithi183 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-alertmanager-a[71868]: ts=2024-03-20T08:48:08.399Z caller=main.go:241 level=info build_context="(go=go1.19.4, user=root@abe866dd5717, date=20221222-14:51:36)" 2024-03-20T08:48:08.683 INFO:journalctl@ceph.alertmanager.a.smithi183.stdout:Mar 20 08:48:08 smithi183 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-alertmanager-a[71868]: ts=2024-03-20T08:48:08.400Z caller=cluster.go:185 level=info component=cluster msg="setting advertise address explicitly" addr=172.21.15.183 port=9094 2024-03-20T08:48:08.683 INFO:journalctl@ceph.alertmanager.a.smithi183.stdout:Mar 20 08:48:08 smithi183 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-alertmanager-a[71868]: ts=2024-03-20T08:48:08.401Z caller=cluster.go:681 level=info component=cluster msg="Waiting for gossip to settle..." interval=2s 2024-03-20T08:48:08.683 INFO:journalctl@ceph.alertmanager.a.smithi183.stdout:Mar 20 08:48:08 smithi183 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-alertmanager-a[71868]: ts=2024-03-20T08:48:08.424Z caller=coordinator.go:113 level=info component=configuration msg="Loading configuration file" file=/etc/alertmanager/alertmanager.yml 2024-03-20T08:48:08.683 INFO:journalctl@ceph.alertmanager.a.smithi183.stdout:Mar 20 08:48:08 smithi183 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-alertmanager-a[71868]: ts=2024-03-20T08:48:08.424Z caller=coordinator.go:126 level=info component=configuration msg="Completed loading of configuration file" file=/etc/alertmanager/alertmanager.yml 2024-03-20T08:48:08.683 INFO:journalctl@ceph.alertmanager.a.smithi183.stdout:Mar 20 08:48:08 smithi183 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-alertmanager-a[71868]: ts=2024-03-20T08:48:08.427Z caller=tls_config.go:232 level=info msg="Listening on" address=[::]:9093 2024-03-20T08:48:08.683 INFO:journalctl@ceph.alertmanager.a.smithi183.stdout:Mar 20 08:48:08 smithi183 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-alertmanager-a[71868]: ts=2024-03-20T08:48:08.427Z caller=tls_config.go:235 level=info msg="TLS is disabled." http2=false address=[::]:9093 2024-03-20T08:48:08.683 INFO:journalctl@ceph.alertmanager.a.smithi183.stdout:Mar 20 08:48:08 smithi183 bash[71841]: 3aa83d988989e0cf6b2a0d2389fcf7b0e8343aad495911a18e5ca7cedfafe732 2024-03-20T08:48:08.683 INFO:journalctl@ceph.alertmanager.a.smithi183.stdout:Mar 20 08:48:08 smithi183 systemd[1]: Started Ceph alertmanager.a for 4c3f3eba-e695-11ee-95c9-87774f69a715. 2024-03-20T08:48:08.813 DEBUG:teuthology.orchestra.run.smithi183:> sudo pkill -f 'journalctl -f -n 0 -u ceph-4c3f3eba-e695-11ee-95c9-87774f69a715@mon.c.service' 2024-03-20T08:48:08.852 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-20T08:48:08.852 INFO:tasks.cephadm.mon.b:Stopped mon.c 2024-03-20T08:48:08.853 INFO:tasks.cephadm.mon.b:Stopping mon.b... 2024-03-20T08:48:08.853 DEBUG:teuthology.orchestra.run.smithi194:> sudo systemctl stop ceph-4c3f3eba-e695-11ee-95c9-87774f69a715@mon.b 2024-03-20T08:48:09.139 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:48:08 smithi194 systemd[1]: Stopping Ceph mon.b for 4c3f3eba-e695-11ee-95c9-87774f69a715... 2024-03-20T08:48:09.476 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:48:09 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-mon-b[27211]: 2024-03-20T08:48:09.165+0000 7f420a137700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.b -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-03-20T08:48:09.476 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:48:09 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-mon-b[27211]: 2024-03-20T08:48:09.165+0000 7f420a137700 -1 mon.b@1(peon) e3 *** Got Signal Terminated *** 2024-03-20T08:48:09.476 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:48:09 smithi194 podman[61349]: 2024-03-20 08:48:09.217902822 +0000 UTC m=+0.167016769 container died da6a20d027a94b6237e2f032b342d32e508375805cb7e427ff891e7d6919fe6a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:976a8b2d06e3c6bfd12488297e8584dd704cc06c3e976c7e0f718c13e238e912, name=ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-mon-b, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, ceph=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-yuri5-testing-2024-03-18-1144-e7e80a6, org.label-schema.build-date=20240307, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS) 2024-03-20T08:48:09.877 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:48:09 smithi194 podman[61349]: 2024-03-20 08:48:09.659033492 +0000 UTC m=+0.608147447 container cleanup da6a20d027a94b6237e2f032b342d32e508375805cb7e427ff891e7d6919fe6a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:976a8b2d06e3c6bfd12488297e8584dd704cc06c3e976c7e0f718c13e238e912, name=ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-mon-b, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, ceph=True, RELEASE=wip-yuri5-testing-2024-03-18-1144-e7e80a6, org.label-schema.build-date=20240307) 2024-03-20T08:48:09.877 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:48:09 smithi194 bash[61349]: ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-mon-b 2024-03-20T08:48:10.128 DEBUG:teuthology.orchestra.run.smithi194:> sudo pkill -f 'journalctl -f -n 0 -u ceph-4c3f3eba-e695-11ee-95c9-87774f69a715@mon.b.service' 2024-03-20T08:48:10.138 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:48:09 smithi194 podman[61363]: 2024-03-20 08:48:09.87757353 +0000 UTC m=+0.657433525 container remove da6a20d027a94b6237e2f032b342d32e508375805cb7e427ff891e7d6919fe6a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:976a8b2d06e3c6bfd12488297e8584dd704cc06c3e976c7e0f718c13e238e912, name=ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-mon-b, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.build-date=20240307, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, ceph=True, maintainer=Guillaume Abrioux , GIT_CLEAN=True, RELEASE=wip-yuri5-testing-2024-03-18-1144-e7e80a6, org.label-schema.schema-version=1.0) 2024-03-20T08:48:10.138 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:48:10 smithi194 systemd[1]: ceph-4c3f3eba-e695-11ee-95c9-87774f69a715@mon.b.service: Deactivated successfully. 2024-03-20T08:48:10.139 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:48:10 smithi194 systemd[1]: Stopped Ceph mon.b for 4c3f3eba-e695-11ee-95c9-87774f69a715. 2024-03-20T08:48:10.139 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 20 08:48:10 smithi194 systemd[1]: ceph-4c3f3eba-e695-11ee-95c9-87774f69a715@mon.b.service: Consumed 5.669s CPU time. 2024-03-20T08:48:10.183 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-20T08:48:10.183 INFO:tasks.cephadm.mon.b:Stopped mon.b 2024-03-20T08:48:10.184 INFO:tasks.cephadm.mgr.y:Stopping mgr.y... 2024-03-20T08:48:10.184 DEBUG:teuthology.orchestra.run.smithi183:> sudo systemctl stop ceph-4c3f3eba-e695-11ee-95c9-87774f69a715@mgr.y 2024-03-20T08:48:10.401 INFO:journalctl@ceph.mgr.y.smithi183.stdout:Mar 20 08:48:10 smithi183 systemd[1]: Stopping Ceph mgr.y for 4c3f3eba-e695-11ee-95c9-87774f69a715... 2024-03-20T08:48:10.704 INFO:journalctl@ceph.alertmanager.a.smithi183.stdout:Mar 20 08:48:10 smithi183 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-alertmanager-a[71868]: ts=2024-03-20T08:48:10.401Z caller=cluster.go:706 level=info component=cluster msg="gossip not settled" polls=0 before=0 now=1 elapsed=2.000154458s 2024-03-20T08:48:11.144 INFO:journalctl@ceph.mgr.y.smithi183.stdout:Mar 20 08:48:10 smithi183 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-mgr-y[25513]: 2024-03-20T08:48:10.827+0000 7fcb17be1700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mgr -n mgr.y -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-20T08:48:11.145 INFO:journalctl@ceph.mgr.y.smithi183.stdout:Mar 20 08:48:10 smithi183 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-mgr-y[25513]: 2024-03-20T08:48:10.827+0000 7fcb17be1700 -1 mgr handle_mgr_signal *** Got signal Terminated *** 2024-03-20T08:48:11.145 INFO:journalctl@ceph.mgr.y.smithi183.stdout:Mar 20 08:48:10 smithi183 podman[72069]: 2024-03-20 08:48:10.859070171 +0000 UTC m=+0.162740085 container died 33072c5d6210372d85db1be2dd63399a8e19a70ce4c0d2dd734057a08ed26f47 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae, name=ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-mgr-y, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-yuri5-testing-2024-03-18-1144-e7e80a6, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, GIT_CLEAN=True, io.buildah.version=1.33.5) 2024-03-20T08:48:11.454 INFO:journalctl@ceph.mgr.y.smithi183.stdout:Mar 20 08:48:11 smithi183 podman[72069]: 2024-03-20 08:48:11.353031204 +0000 UTC m=+0.656701123 container cleanup 33072c5d6210372d85db1be2dd63399a8e19a70ce4c0d2dd734057a08ed26f47 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae, name=ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-mgr-y, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, ceph=True, RELEASE=wip-yuri5-testing-2024-03-18-1144-e7e80a6, GIT_CLEAN=True, io.buildah.version=1.33.5) 2024-03-20T08:48:11.454 INFO:journalctl@ceph.mgr.y.smithi183.stdout:Mar 20 08:48:11 smithi183 bash[72069]: ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-mgr-y 2024-03-20T08:48:11.783 DEBUG:teuthology.orchestra.run.smithi183:> sudo pkill -f 'journalctl -f -n 0 -u ceph-4c3f3eba-e695-11ee-95c9-87774f69a715@mgr.y.service' 2024-03-20T08:48:11.822 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-20T08:48:11.822 INFO:tasks.cephadm.mgr.y:Stopped mgr.y 2024-03-20T08:48:11.822 INFO:tasks.cephadm.mgr.x:Stopping mgr.x... 2024-03-20T08:48:11.822 DEBUG:teuthology.orchestra.run.smithi194:> sudo systemctl stop ceph-4c3f3eba-e695-11ee-95c9-87774f69a715@mgr.x 2024-03-20T08:48:12.091 INFO:journalctl@ceph.mgr.x.smithi194.stdout:Mar 20 08:48:11 smithi194 systemd[1]: Stopping Ceph mgr.x for 4c3f3eba-e695-11ee-95c9-87774f69a715... 2024-03-20T08:48:12.388 INFO:journalctl@ceph.mgr.x.smithi194.stdout:Mar 20 08:48:12 smithi194 podman[61470]: 2024-03-20 08:48:12.171284197 +0000 UTC m=+0.154651824 container died 625764fcda26def3fba993831c484f9fe43d54b4951ee495cd88855b47232a6e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:976a8b2d06e3c6bfd12488297e8584dd704cc06c3e976c7e0f718c13e238e912, name=ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-mgr-x, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-yuri5-testing-2024-03-18-1144-e7e80a6, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, ceph=True, CEPH_POINT_RELEASE=) 2024-03-20T08:48:12.807 INFO:journalctl@ceph.mgr.x.smithi194.stdout:Mar 20 08:48:12 smithi194 podman[61470]: 2024-03-20 08:48:12.522542386 +0000 UTC m=+0.505910025 container cleanup 625764fcda26def3fba993831c484f9fe43d54b4951ee495cd88855b47232a6e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:976a8b2d06e3c6bfd12488297e8584dd704cc06c3e976c7e0f718c13e238e912, name=ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-mgr-x, org.label-schema.build-date=20240307, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-yuri5-testing-2024-03-18-1144-e7e80a6, ceph=True, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-20T08:48:12.807 INFO:journalctl@ceph.mgr.x.smithi194.stdout:Mar 20 08:48:12 smithi194 bash[61470]: ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-mgr-x 2024-03-20T08:48:12.943 DEBUG:teuthology.orchestra.run.smithi194:> sudo pkill -f 'journalctl -f -n 0 -u ceph-4c3f3eba-e695-11ee-95c9-87774f69a715@mgr.x.service' 2024-03-20T08:48:12.982 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-20T08:48:12.982 INFO:tasks.cephadm.mgr.x:Stopped mgr.x 2024-03-20T08:48:12.982 INFO:tasks.cephadm.osd.0:Stopping osd.0... 2024-03-20T08:48:12.982 DEBUG:teuthology.orchestra.run.smithi183:> sudo systemctl stop ceph-4c3f3eba-e695-11ee-95c9-87774f69a715@osd.0 2024-03-20T08:48:13.271 INFO:journalctl@ceph.osd.0.smithi183.stdout:Mar 20 08:48:13 smithi183 systemd[1]: Stopping Ceph osd.0 for 4c3f3eba-e695-11ee-95c9-87774f69a715... 2024-03-20T08:48:13.704 INFO:journalctl@ceph.osd.0.smithi183.stdout:Mar 20 08:48:13 smithi183 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-osd-0[36374]: 2024-03-20T08:48:13.299+0000 7f89c5956700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.0 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-20T08:48:13.704 INFO:journalctl@ceph.osd.0.smithi183.stdout:Mar 20 08:48:13 smithi183 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-osd-0[36374]: 2024-03-20T08:48:13.299+0000 7f89c5956700 -1 osd.0 71 *** Got signal Terminated *** 2024-03-20T08:48:13.704 INFO:journalctl@ceph.osd.0.smithi183.stdout:Mar 20 08:48:13 smithi183 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-osd-0[36374]: 2024-03-20T08:48:13.299+0000 7f89c5956700 -1 osd.0 71 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-20T08:48:18.704 INFO:journalctl@ceph.alertmanager.a.smithi183.stdout:Mar 20 08:48:18 smithi183 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-alertmanager-a[71868]: ts=2024-03-20T08:48:18.404Z caller=cluster.go:698 level=info component=cluster msg="gossip settled; proceeding" elapsed=10.002369651s 2024-03-20T08:48:19.024 INFO:journalctl@ceph.osd.0.smithi183.stdout:Mar 20 08:48:18 smithi183 podman[72191]: 2024-03-20 08:48:18.742003062 +0000 UTC m=+5.560079633 container died e86ec9850e30e44ee58303ab0fd54499a4d42d544f78d6b899e349c3c8f43983 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:976a8b2d06e3c6bfd12488297e8584dd704cc06c3e976c7e0f718c13e238e912, name=ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-osd-0, ceph=True, maintainer=Guillaume Abrioux , RELEASE=wip-yuri5-testing-2024-03-18-1144-e7e80a6, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240307, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2) 2024-03-20T08:48:19.454 INFO:journalctl@ceph.osd.0.smithi183.stdout:Mar 20 08:48:19 smithi183 podman[72191]: 2024-03-20 08:48:19.198768012 +0000 UTC m=+6.016844583 container cleanup e86ec9850e30e44ee58303ab0fd54499a4d42d544f78d6b899e349c3c8f43983 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:976a8b2d06e3c6bfd12488297e8584dd704cc06c3e976c7e0f718c13e238e912, name=ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-osd-0, ceph=True, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, RELEASE=wip-yuri5-testing-2024-03-18-1144-e7e80a6, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240307) 2024-03-20T08:48:19.455 INFO:journalctl@ceph.osd.0.smithi183.stdout:Mar 20 08:48:19 smithi183 bash[72191]: ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-osd-0 2024-03-20T08:48:19.784 INFO:journalctl@ceph.osd.0.smithi183.stdout:Mar 20 08:48:19 smithi183 podman[72477]: 2024-03-20 08:48:19.466846515 +0000 UTC m=+0.726630574 container remove e86ec9850e30e44ee58303ab0fd54499a4d42d544f78d6b899e349c3c8f43983 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:976a8b2d06e3c6bfd12488297e8584dd704cc06c3e976c7e0f718c13e238e912, name=ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-osd-0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, maintainer=Guillaume Abrioux , RELEASE=wip-yuri5-testing-2024-03-18-1144-e7e80a6, org.label-schema.build-date=20240307, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2) 2024-03-20T08:48:20.764 INFO:journalctl@ceph.osd.0.smithi183.stdout:Mar 20 08:48:20 smithi183 podman[72698]: 2024-03-20 08:48:20.462162527 +0000 UTC m=+0.114440665 container create ee4545bc5981cf901daeea1c21baa3349dcec7e1abf08bab58992756f76e5947 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:976a8b2d06e3c6bfd12488297e8584dd704cc06c3e976c7e0f718c13e238e912, name=ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-osd-0-deactivate, ceph=True, GIT_CLEAN=True, RELEASE=wip-yuri5-testing-2024-03-18-1144-e7e80a6, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240307, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0) 2024-03-20T08:48:20.764 INFO:journalctl@ceph.osd.0.smithi183.stdout:Mar 20 08:48:20 smithi183 podman[72698]: 2024-03-20 08:48:20.366193309 +0000 UTC m=+0.018471460 image pull 9ceba81fb2c8d8682cb159527017d591637b153b3aa4a390ffa8586db6dcda00 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:976a8b2d06e3c6bfd12488297e8584dd704cc06c3e976c7e0f718c13e238e912 2024-03-20T08:48:20.764 INFO:journalctl@ceph.osd.0.smithi183.stdout:Mar 20 08:48:20 smithi183 podman[72698]: 2024-03-20 08:48:20.689445625 +0000 UTC m=+0.341723764 container init ee4545bc5981cf901daeea1c21baa3349dcec7e1abf08bab58992756f76e5947 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:976a8b2d06e3c6bfd12488297e8584dd704cc06c3e976c7e0f718c13e238e912, name=ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-osd-0-deactivate, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-yuri5-testing-2024-03-18-1144-e7e80a6, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.build-date=20240307, ceph=True, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0) 2024-03-20T08:48:20.764 INFO:journalctl@ceph.osd.0.smithi183.stdout:Mar 20 08:48:20 smithi183 podman[72698]: 2024-03-20 08:48:20.693508452 +0000 UTC m=+0.345786596 container start ee4545bc5981cf901daeea1c21baa3349dcec7e1abf08bab58992756f76e5947 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:976a8b2d06e3c6bfd12488297e8584dd704cc06c3e976c7e0f718c13e238e912, name=ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-osd-0-deactivate, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-yuri5-testing-2024-03-18-1144-e7e80a6, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240307, ceph=True, org.label-schema.license=GPLv2) 2024-03-20T08:48:20.764 INFO:journalctl@ceph.osd.0.smithi183.stdout:Mar 20 08:48:20 smithi183 podman[72698]: 2024-03-20 08:48:20.764453739 +0000 UTC m=+0.416731884 container attach ee4545bc5981cf901daeea1c21baa3349dcec7e1abf08bab58992756f76e5947 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:976a8b2d06e3c6bfd12488297e8584dd704cc06c3e976c7e0f718c13e238e912, name=ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-osd-0-deactivate, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.build-date=20240307, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-yuri5-testing-2024-03-18-1144-e7e80a6, org.label-schema.vendor=CentOS, ceph=True, GIT_CLEAN=True) 2024-03-20T08:48:21.445 INFO:journalctl@ceph.prometheus.a.smithi194.stdout:Mar 20 08:48:21 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-prometheus-a[49402]: ts=2024-03-20T08:48:21.297Z caller=refresh.go:99 level=error component="discovery manager scrape" discovery=http config=ceph msg="Unable to refresh target groups" err="Get \"http://172.21.15.183:8765/sd/prometheus/sd-config?service=mgr-prometheus\": dial tcp 172.21.15.183:8765: connect: connection refused" 2024-03-20T08:48:21.454 INFO:journalctl@ceph.osd.0.smithi183.stdout:Mar 20 08:48:21 smithi183 podman[72698]: 2024-03-20 08:48:21.063690081 +0000 UTC m=+0.715968228 container died ee4545bc5981cf901daeea1c21baa3349dcec7e1abf08bab58992756f76e5947 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:976a8b2d06e3c6bfd12488297e8584dd704cc06c3e976c7e0f718c13e238e912, name=ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-osd-0-deactivate, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240307, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, RELEASE=wip-yuri5-testing-2024-03-18-1144-e7e80a6, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, ceph=True, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-20T08:48:21.716 DEBUG:teuthology.orchestra.run.smithi183:> sudo pkill -f 'journalctl -f -n 0 -u ceph-4c3f3eba-e695-11ee-95c9-87774f69a715@osd.0.service' 2024-03-20T08:48:21.744 INFO:journalctl@ceph.osd.0.smithi183.stdout:Mar 20 08:48:21 smithi183 podman[72782]: 2024-03-20 08:48:21.694538125 +0000 UTC m=+0.704408086 container remove ee4545bc5981cf901daeea1c21baa3349dcec7e1abf08bab58992756f76e5947 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:976a8b2d06e3c6bfd12488297e8584dd704cc06c3e976c7e0f718c13e238e912, name=ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-osd-0-deactivate, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, org.label-schema.build-date=20240307, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , RELEASE=wip-yuri5-testing-2024-03-18-1144-e7e80a6, GIT_BRANCH=HEAD) 2024-03-20T08:48:21.744 INFO:journalctl@ceph.osd.0.smithi183.stdout:Mar 20 08:48:21 smithi183 systemd[1]: ceph-4c3f3eba-e695-11ee-95c9-87774f69a715@osd.0.service: Deactivated successfully. 2024-03-20T08:48:21.744 INFO:journalctl@ceph.osd.0.smithi183.stdout:Mar 20 08:48:21 smithi183 systemd[1]: Stopped Ceph osd.0 for 4c3f3eba-e695-11ee-95c9-87774f69a715. 2024-03-20T08:48:21.744 INFO:journalctl@ceph.osd.0.smithi183.stdout:Mar 20 08:48:21 smithi183 systemd[1]: ceph-4c3f3eba-e695-11ee-95c9-87774f69a715@osd.0.service: Consumed 3.421s CPU time. 2024-03-20T08:48:21.761 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-20T08:48:21.761 INFO:tasks.cephadm.osd.0:Stopped osd.0 2024-03-20T08:48:21.762 INFO:tasks.cephadm.osd.1:Stopping osd.1... 2024-03-20T08:48:21.762 DEBUG:teuthology.orchestra.run.smithi183:> sudo systemctl stop ceph-4c3f3eba-e695-11ee-95c9-87774f69a715@osd.1 2024-03-20T08:48:22.181 INFO:journalctl@ceph.osd.1.smithi183.stdout:Mar 20 08:48:21 smithi183 systemd[1]: Stopping Ceph osd.1 for 4c3f3eba-e695-11ee-95c9-87774f69a715... 2024-03-20T08:48:23.899 INFO:journalctl@ceph.osd.1.smithi183.stdout:Mar 20 08:48:23 smithi183 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-osd-1[39792]: 2024-03-20T08:48:23.799+0000 7f8e975c2700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.1 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-20T08:48:23.900 INFO:journalctl@ceph.osd.1.smithi183.stdout:Mar 20 08:48:23 smithi183 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-osd-1[39792]: 2024-03-20T08:48:23.799+0000 7f8e975c2700 -1 osd.1 71 *** Got signal Terminated *** 2024-03-20T08:48:23.900 INFO:journalctl@ceph.osd.1.smithi183.stdout:Mar 20 08:48:23 smithi183 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-osd-1[39792]: 2024-03-20T08:48:23.799+0000 7f8e975c2700 -1 osd.1 71 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-20T08:48:29.422 INFO:journalctl@ceph.osd.1.smithi183.stdout:Mar 20 08:48:29 smithi183 podman[72936]: 2024-03-20 08:48:29.112831357 +0000 UTC m=+6.796141618 container died 93c340a16ba3acaf9029892b4c10e4762fe87e8392c7910ad5db005d02b490b5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:976a8b2d06e3c6bfd12488297e8584dd704cc06c3e976c7e0f718c13e238e912, name=ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-osd-1, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_CLEAN=True, ceph=True, org.label-schema.build-date=20240307, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, RELEASE=wip-yuri5-testing-2024-03-18-1144-e7e80a6, org.label-schema.license=GPLv2) 2024-03-20T08:48:29.704 INFO:journalctl@ceph.osd.1.smithi183.stdout:Mar 20 08:48:29 smithi183 podman[72936]: 2024-03-20 08:48:29.605403666 +0000 UTC m=+7.288713926 container cleanup 93c340a16ba3acaf9029892b4c10e4762fe87e8392c7910ad5db005d02b490b5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:976a8b2d06e3c6bfd12488297e8584dd704cc06c3e976c7e0f718c13e238e912, name=ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-osd-1, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, ceph=True, org.label-schema.license=GPLv2, org.label-schema.build-date=20240307, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_CLEAN=True, RELEASE=wip-yuri5-testing-2024-03-18-1144-e7e80a6, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=) 2024-03-20T08:48:29.704 INFO:journalctl@ceph.osd.1.smithi183.stdout:Mar 20 08:48:29 smithi183 bash[72936]: ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-osd-1 2024-03-20T08:48:30.142 INFO:journalctl@ceph.osd.1.smithi183.stdout:Mar 20 08:48:29 smithi183 podman[73118]: 2024-03-20 08:48:29.814966785 +0000 UTC m=+0.704502096 container remove 93c340a16ba3acaf9029892b4c10e4762fe87e8392c7910ad5db005d02b490b5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:976a8b2d06e3c6bfd12488297e8584dd704cc06c3e976c7e0f718c13e238e912, name=ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-osd-1, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240307, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, RELEASE=wip-yuri5-testing-2024-03-18-1144-e7e80a6, ceph=True) 2024-03-20T08:48:30.409 INFO:journalctl@ceph.osd.1.smithi183.stdout:Mar 20 08:48:30 smithi183 podman[73212]: 2024-03-20 08:48:30.046556742 +0000 UTC m=+0.018127902 image pull 9ceba81fb2c8d8682cb159527017d591637b153b3aa4a390ffa8586db6dcda00 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:976a8b2d06e3c6bfd12488297e8584dd704cc06c3e976c7e0f718c13e238e912 2024-03-20T08:48:30.409 INFO:journalctl@ceph.osd.1.smithi183.stdout:Mar 20 08:48:30 smithi183 podman[73212]: 2024-03-20 08:48:30.157497376 +0000 UTC m=+0.129068525 container create 2221006f1f6e792ed6095798e8b3a1b527ea05183d0ffb8b2604ef00f7115ede (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:976a8b2d06e3c6bfd12488297e8584dd704cc06c3e976c7e0f718c13e238e912, name=ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-osd-1-deactivate, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.build-date=20240307, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-yuri5-testing-2024-03-18-1144-e7e80a6, org.label-schema.license=GPLv2, io.buildah.version=1.33.5) 2024-03-20T08:48:30.704 INFO:journalctl@ceph.osd.1.smithi183.stdout:Mar 20 08:48:30 smithi183 podman[73212]: 2024-03-20 08:48:30.409811274 +0000 UTC m=+0.381382439 container init 2221006f1f6e792ed6095798e8b3a1b527ea05183d0ffb8b2604ef00f7115ede (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:976a8b2d06e3c6bfd12488297e8584dd704cc06c3e976c7e0f718c13e238e912, name=ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-osd-1-deactivate, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.build-date=20240307, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, RELEASE=wip-yuri5-testing-2024-03-18-1144-e7e80a6, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux ) 2024-03-20T08:48:30.704 INFO:journalctl@ceph.osd.1.smithi183.stdout:Mar 20 08:48:30 smithi183 podman[73212]: 2024-03-20 08:48:30.412980695 +0000 UTC m=+0.384551855 container start 2221006f1f6e792ed6095798e8b3a1b527ea05183d0ffb8b2604ef00f7115ede (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:976a8b2d06e3c6bfd12488297e8584dd704cc06c3e976c7e0f718c13e238e912, name=ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-osd-1-deactivate, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240307, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, RELEASE=wip-yuri5-testing-2024-03-18-1144-e7e80a6, org.label-schema.license=GPLv2) 2024-03-20T08:48:30.705 INFO:journalctl@ceph.osd.1.smithi183.stdout:Mar 20 08:48:30 smithi183 podman[73212]: 2024-03-20 08:48:30.509727557 +0000 UTC m=+0.481298723 container attach 2221006f1f6e792ed6095798e8b3a1b527ea05183d0ffb8b2604ef00f7115ede (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:976a8b2d06e3c6bfd12488297e8584dd704cc06c3e976c7e0f718c13e238e912, name=ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-osd-1-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240307, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, RELEASE=wip-yuri5-testing-2024-03-18-1144-e7e80a6, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0) 2024-03-20T08:48:31.204 INFO:journalctl@ceph.osd.1.smithi183.stdout:Mar 20 08:48:30 smithi183 podman[73212]: 2024-03-20 08:48:30.711116966 +0000 UTC m=+0.682688122 container died 2221006f1f6e792ed6095798e8b3a1b527ea05183d0ffb8b2604ef00f7115ede (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:976a8b2d06e3c6bfd12488297e8584dd704cc06c3e976c7e0f718c13e238e912, name=ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-osd-1-deactivate, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, RELEASE=wip-yuri5-testing-2024-03-18-1144-e7e80a6, maintainer=Guillaume Abrioux ) 2024-03-20T08:48:31.296 DEBUG:teuthology.orchestra.run.smithi183:> sudo pkill -f 'journalctl -f -n 0 -u ceph-4c3f3eba-e695-11ee-95c9-87774f69a715@osd.1.service' 2024-03-20T08:48:31.337 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-20T08:48:31.338 INFO:tasks.cephadm.osd.1:Stopped osd.1 2024-03-20T08:48:31.338 INFO:tasks.cephadm.osd.2:Stopping osd.2... 2024-03-20T08:48:31.338 DEBUG:teuthology.orchestra.run.smithi183:> sudo systemctl stop ceph-4c3f3eba-e695-11ee-95c9-87774f69a715@osd.2 2024-03-20T08:48:31.550 INFO:journalctl@ceph.osd.2.smithi183.stdout:Mar 20 08:48:31 smithi183 systemd[1]: Stopping Ceph osd.2 for 4c3f3eba-e695-11ee-95c9-87774f69a715... 2024-03-20T08:48:31.954 INFO:journalctl@ceph.osd.2.smithi183.stdout:Mar 20 08:48:31 smithi183 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-osd-2[43333]: 2024-03-20T08:48:31.707+0000 7f8d14427700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.2 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-20T08:48:31.955 INFO:journalctl@ceph.osd.2.smithi183.stdout:Mar 20 08:48:31 smithi183 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-osd-2[43333]: 2024-03-20T08:48:31.707+0000 7f8d14427700 -1 osd.2 71 *** Got signal Terminated *** 2024-03-20T08:48:31.955 INFO:journalctl@ceph.osd.2.smithi183.stdout:Mar 20 08:48:31 smithi183 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-osd-2[43333]: 2024-03-20T08:48:31.707+0000 7f8d14427700 -1 osd.2 71 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-20T08:48:37.416 INFO:journalctl@ceph.osd.2.smithi183.stdout:Mar 20 08:48:37 smithi183 podman[73317]: 2024-03-20 08:48:37.141860409 +0000 UTC m=+5.559606913 container died db171b2995f22771910c297c49da6b1afeac331de21f04a2c2fa3ec36f12b09f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:976a8b2d06e3c6bfd12488297e8584dd704cc06c3e976c7e0f718c13e238e912, name=ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-osd-2, RELEASE=wip-yuri5-testing-2024-03-18-1144-e7e80a6, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, ceph=True, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240307, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux ) 2024-03-20T08:48:37.704 INFO:journalctl@ceph.osd.2.smithi183.stdout:Mar 20 08:48:37 smithi183 podman[73317]: 2024-03-20 08:48:37.59898612 +0000 UTC m=+6.016732632 container cleanup db171b2995f22771910c297c49da6b1afeac331de21f04a2c2fa3ec36f12b09f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:976a8b2d06e3c6bfd12488297e8584dd704cc06c3e976c7e0f718c13e238e912, name=ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-osd-2, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240307, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-yuri5-testing-2024-03-18-1144-e7e80a6, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, ceph=True, GIT_CLEAN=True, io.buildah.version=1.33.5) 2024-03-20T08:48:37.704 INFO:journalctl@ceph.osd.2.smithi183.stdout:Mar 20 08:48:37 smithi183 bash[73317]: ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-osd-2 2024-03-20T08:48:38.204 INFO:journalctl@ceph.osd.2.smithi183.stdout:Mar 20 08:48:37 smithi183 podman[73499]: 2024-03-20 08:48:37.916763941 +0000 UTC m=+0.777663225 container remove db171b2995f22771910c297c49da6b1afeac331de21f04a2c2fa3ec36f12b09f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:976a8b2d06e3c6bfd12488297e8584dd704cc06c3e976c7e0f718c13e238e912, name=ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-osd-2, GIT_BRANCH=HEAD, ceph=True, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240307, maintainer=Guillaume Abrioux , RELEASE=wip-yuri5-testing-2024-03-18-1144-e7e80a6, org.label-schema.schema-version=1.0) 2024-03-20T08:48:38.661 INFO:journalctl@ceph.osd.2.smithi183.stdout:Mar 20 08:48:38 smithi183 podman[73635]: 2024-03-20 08:48:38.207256646 +0000 UTC m=+0.018515437 image pull 9ceba81fb2c8d8682cb159527017d591637b153b3aa4a390ffa8586db6dcda00 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:976a8b2d06e3c6bfd12488297e8584dd704cc06c3e976c7e0f718c13e238e912 2024-03-20T08:48:38.661 INFO:journalctl@ceph.osd.2.smithi183.stdout:Mar 20 08:48:38 smithi183 podman[73635]: 2024-03-20 08:48:38.325847497 +0000 UTC m=+0.137106280 container create b71927bd1ce23ce93d12fffc4939d17874693cb93b4b5a5bbdd7e977397c5d5a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:976a8b2d06e3c6bfd12488297e8584dd704cc06c3e976c7e0f718c13e238e912, name=ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-osd-2-deactivate, org.label-schema.build-date=20240307, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , RELEASE=wip-yuri5-testing-2024-03-18-1144-e7e80a6, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, ceph=True, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-20T08:48:38.661 INFO:journalctl@ceph.osd.2.smithi183.stdout:Mar 20 08:48:38 smithi183 podman[73635]: 2024-03-20 08:48:38.536323795 +0000 UTC m=+0.347582578 container init b71927bd1ce23ce93d12fffc4939d17874693cb93b4b5a5bbdd7e977397c5d5a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:976a8b2d06e3c6bfd12488297e8584dd704cc06c3e976c7e0f718c13e238e912, name=ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-osd-2-deactivate, RELEASE=wip-yuri5-testing-2024-03-18-1144-e7e80a6, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_CLEAN=True) 2024-03-20T08:48:38.661 INFO:journalctl@ceph.osd.2.smithi183.stdout:Mar 20 08:48:38 smithi183 podman[73635]: 2024-03-20 08:48:38.540223934 +0000 UTC m=+0.351482718 container start b71927bd1ce23ce93d12fffc4939d17874693cb93b4b5a5bbdd7e977397c5d5a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:976a8b2d06e3c6bfd12488297e8584dd704cc06c3e976c7e0f718c13e238e912, name=ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-osd-2-deactivate, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.build-date=20240307, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, RELEASE=wip-yuri5-testing-2024-03-18-1144-e7e80a6, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-20T08:48:38.661 INFO:journalctl@ceph.osd.2.smithi183.stdout:Mar 20 08:48:38 smithi183 podman[73635]: 2024-03-20 08:48:38.661237249 +0000 UTC m=+0.472496053 container attach b71927bd1ce23ce93d12fffc4939d17874693cb93b4b5a5bbdd7e977397c5d5a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:976a8b2d06e3c6bfd12488297e8584dd704cc06c3e976c7e0f718c13e238e912, name=ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-osd-2-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.build-date=20240307, RELEASE=wip-yuri5-testing-2024-03-18-1144-e7e80a6, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, ceph=True) 2024-03-20T08:48:38.954 INFO:journalctl@ceph.osd.2.smithi183.stdout:Mar 20 08:48:38 smithi183 podman[73635]: 2024-03-20 08:48:38.860313913 +0000 UTC m=+0.671572712 container died b71927bd1ce23ce93d12fffc4939d17874693cb93b4b5a5bbdd7e977397c5d5a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:976a8b2d06e3c6bfd12488297e8584dd704cc06c3e976c7e0f718c13e238e912, name=ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-osd-2-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , ceph=True, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240307, RELEASE=wip-yuri5-testing-2024-03-18-1144-e7e80a6, org.label-schema.license=GPLv2, GIT_CLEAN=True) 2024-03-20T08:48:39.512 DEBUG:teuthology.orchestra.run.smithi183:> sudo pkill -f 'journalctl -f -n 0 -u ceph-4c3f3eba-e695-11ee-95c9-87774f69a715@osd.2.service' 2024-03-20T08:48:39.557 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-20T08:48:39.558 INFO:tasks.cephadm.osd.2:Stopped osd.2 2024-03-20T08:48:39.558 INFO:tasks.cephadm.osd.3:Stopping osd.3... 2024-03-20T08:48:39.558 DEBUG:teuthology.orchestra.run.smithi183:> sudo systemctl stop ceph-4c3f3eba-e695-11ee-95c9-87774f69a715@osd.3 2024-03-20T08:48:39.859 INFO:journalctl@ceph.osd.3.smithi183.stdout:Mar 20 08:48:39 smithi183 systemd[1]: Stopping Ceph osd.3 for 4c3f3eba-e695-11ee-95c9-87774f69a715... 2024-03-20T08:48:40.923 INFO:journalctl@ceph.osd.3.smithi183.stdout:Mar 20 08:48:40 smithi183 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-osd-3[47180]: 2024-03-20T08:48:40.840+0000 7f54388b1700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.3 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-20T08:48:40.923 INFO:journalctl@ceph.osd.3.smithi183.stdout:Mar 20 08:48:40 smithi183 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-osd-3[47180]: 2024-03-20T08:48:40.840+0000 7f54388b1700 -1 osd.3 71 *** Got signal Terminated *** 2024-03-20T08:48:40.923 INFO:journalctl@ceph.osd.3.smithi183.stdout:Mar 20 08:48:40 smithi183 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-osd-3[47180]: 2024-03-20T08:48:40.840+0000 7f54388b1700 -1 osd.3 71 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-20T08:48:41.205 INFO:journalctl@ceph.osd.3.smithi183.stdout:Mar 20 08:48:41 smithi183 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-osd-3[47180]: 2024-03-20T08:48:41.043+0000 7f5430e9d700 -1 osd.3 71 heartbeat_check: no reply from 172.21.15.183:6806 osd.0 since back 2024-03-20T08:48:15.142331+0000 front 2024-03-20T08:48:15.142245+0000 (oldest deadline 2024-03-20T08:48:41.042229+0000) 2024-03-20T08:48:41.241 INFO:journalctl@ceph.osd.4.smithi194.stdout:Mar 20 08:48:41 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-osd-4[34266]: 2024-03-20T08:48:41.171+0000 7fd5262d1700 -1 osd.4 71 heartbeat_check: no reply from 172.21.15.183:6806 osd.0 since back 2024-03-20T08:48:16.599224+0000 front 2024-03-20T08:48:16.599321+0000 (oldest deadline 2024-03-20T08:48:40.699108+0000) 2024-03-20T08:48:41.576 INFO:journalctl@ceph.osd.6.smithi194.stdout:Mar 20 08:48:41 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-osd-6[41285]: 2024-03-20T08:48:41.441+0000 7fabb4d64700 -1 osd.6 71 heartbeat_check: no reply from 172.21.15.183:6806 osd.0 since back 2024-03-20T08:48:18.168836+0000 front 2024-03-20T08:48:18.168757+0000 (oldest deadline 2024-03-20T08:48:40.468676+0000) 2024-03-20T08:48:41.889 INFO:journalctl@ceph.osd.5.smithi194.stdout:Mar 20 08:48:41 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-osd-5[37760]: 2024-03-20T08:48:41.575+0000 7f60920aa700 -1 osd.5 71 heartbeat_check: no reply from 172.21.15.183:6806 osd.0 since back 2024-03-20T08:48:15.356781+0000 front 2024-03-20T08:48:15.356583+0000 (oldest deadline 2024-03-20T08:48:40.656479+0000) 2024-03-20T08:48:42.378 INFO:journalctl@ceph.osd.3.smithi183.stdout:Mar 20 08:48:42 smithi183 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-osd-3[47180]: 2024-03-20T08:48:42.043+0000 7f5430e9d700 -1 osd.3 71 heartbeat_check: no reply from 172.21.15.183:6806 osd.0 since back 2024-03-20T08:48:15.142331+0000 front 2024-03-20T08:48:15.142245+0000 (oldest deadline 2024-03-20T08:48:41.042229+0000) 2024-03-20T08:48:42.480 INFO:journalctl@ceph.osd.6.smithi194.stdout:Mar 20 08:48:42 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-osd-6[41285]: 2024-03-20T08:48:42.429+0000 7fabb4d64700 -1 osd.6 71 heartbeat_check: no reply from 172.21.15.183:6806 osd.0 since back 2024-03-20T08:48:18.168836+0000 front 2024-03-20T08:48:18.168757+0000 (oldest deadline 2024-03-20T08:48:40.468676+0000) 2024-03-20T08:48:42.480 INFO:journalctl@ceph.osd.4.smithi194.stdout:Mar 20 08:48:42 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-osd-4[34266]: 2024-03-20T08:48:42.183+0000 7fd5262d1700 -1 osd.4 71 heartbeat_check: no reply from 172.21.15.183:6806 osd.0 since back 2024-03-20T08:48:16.599224+0000 front 2024-03-20T08:48:16.599321+0000 (oldest deadline 2024-03-20T08:48:40.699108+0000) 2024-03-20T08:48:42.888 INFO:journalctl@ceph.osd.5.smithi194.stdout:Mar 20 08:48:42 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-osd-5[37760]: 2024-03-20T08:48:42.621+0000 7f60920aa700 -1 osd.5 71 heartbeat_check: no reply from 172.21.15.183:6806 osd.0 since back 2024-03-20T08:48:15.356781+0000 front 2024-03-20T08:48:15.356583+0000 (oldest deadline 2024-03-20T08:48:40.656479+0000) 2024-03-20T08:48:42.889 INFO:journalctl@ceph.osd.7.smithi194.stdout:Mar 20 08:48:42 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-osd-7[44872]: 2024-03-20T08:48:42.602+0000 7f37d5961700 -1 osd.7 71 heartbeat_check: no reply from 172.21.15.183:6806 osd.0 since back 2024-03-20T08:48:16.746885+0000 front 2024-03-20T08:48:16.746994+0000 (oldest deadline 2024-03-20T08:48:42.046867+0000) 2024-03-20T08:48:43.454 INFO:journalctl@ceph.osd.3.smithi183.stdout:Mar 20 08:48:43 smithi183 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-osd-3[47180]: 2024-03-20T08:48:43.007+0000 7f5430e9d700 -1 osd.3 71 heartbeat_check: no reply from 172.21.15.183:6806 osd.0 since back 2024-03-20T08:48:15.142331+0000 front 2024-03-20T08:48:15.142245+0000 (oldest deadline 2024-03-20T08:48:41.042229+0000) 2024-03-20T08:48:43.604 INFO:journalctl@ceph.osd.4.smithi194.stdout:Mar 20 08:48:43 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-osd-4[34266]: 2024-03-20T08:48:43.218+0000 7fd5262d1700 -1 osd.4 71 heartbeat_check: no reply from 172.21.15.183:6806 osd.0 since back 2024-03-20T08:48:16.599224+0000 front 2024-03-20T08:48:16.599321+0000 (oldest deadline 2024-03-20T08:48:40.699108+0000) 2024-03-20T08:48:43.604 INFO:journalctl@ceph.osd.6.smithi194.stdout:Mar 20 08:48:43 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-osd-6[41285]: 2024-03-20T08:48:43.452+0000 7fabb4d64700 -1 osd.6 71 heartbeat_check: no reply from 172.21.15.183:6806 osd.0 since back 2024-03-20T08:48:18.168836+0000 front 2024-03-20T08:48:18.168757+0000 (oldest deadline 2024-03-20T08:48:40.468676+0000) 2024-03-20T08:48:43.888 INFO:journalctl@ceph.osd.5.smithi194.stdout:Mar 20 08:48:43 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-osd-5[37760]: 2024-03-20T08:48:43.603+0000 7f60920aa700 -1 osd.5 71 heartbeat_check: no reply from 172.21.15.183:6806 osd.0 since back 2024-03-20T08:48:15.356781+0000 front 2024-03-20T08:48:15.356583+0000 (oldest deadline 2024-03-20T08:48:40.656479+0000) 2024-03-20T08:48:43.889 INFO:journalctl@ceph.osd.7.smithi194.stdout:Mar 20 08:48:43 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-osd-7[44872]: 2024-03-20T08:48:43.641+0000 7f37d5961700 -1 osd.7 71 heartbeat_check: no reply from 172.21.15.183:6806 osd.0 since back 2024-03-20T08:48:16.746885+0000 front 2024-03-20T08:48:16.746994+0000 (oldest deadline 2024-03-20T08:48:42.046867+0000) 2024-03-20T08:48:44.454 INFO:journalctl@ceph.osd.3.smithi183.stdout:Mar 20 08:48:44 smithi183 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-osd-3[47180]: 2024-03-20T08:48:44.025+0000 7f5430e9d700 -1 osd.3 71 heartbeat_check: no reply from 172.21.15.183:6806 osd.0 since back 2024-03-20T08:48:15.142331+0000 front 2024-03-20T08:48:15.142245+0000 (oldest deadline 2024-03-20T08:48:41.042229+0000) 2024-03-20T08:48:44.589 INFO:journalctl@ceph.osd.4.smithi194.stdout:Mar 20 08:48:44 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-osd-4[34266]: 2024-03-20T08:48:44.259+0000 7fd5262d1700 -1 osd.4 71 heartbeat_check: no reply from 172.21.15.183:6806 osd.0 since back 2024-03-20T08:48:16.599224+0000 front 2024-03-20T08:48:16.599321+0000 (oldest deadline 2024-03-20T08:48:40.699108+0000) 2024-03-20T08:48:44.589 INFO:journalctl@ceph.osd.6.smithi194.stdout:Mar 20 08:48:44 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-osd-6[41285]: 2024-03-20T08:48:44.454+0000 7fabb4d64700 -1 osd.6 71 heartbeat_check: no reply from 172.21.15.183:6806 osd.0 since back 2024-03-20T08:48:18.168836+0000 front 2024-03-20T08:48:18.168757+0000 (oldest deadline 2024-03-20T08:48:40.468676+0000) 2024-03-20T08:48:44.888 INFO:journalctl@ceph.osd.5.smithi194.stdout:Mar 20 08:48:44 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-osd-5[37760]: 2024-03-20T08:48:44.589+0000 7f60920aa700 -1 osd.5 71 heartbeat_check: no reply from 172.21.15.183:6806 osd.0 since back 2024-03-20T08:48:15.356781+0000 front 2024-03-20T08:48:15.356583+0000 (oldest deadline 2024-03-20T08:48:40.656479+0000) 2024-03-20T08:48:44.889 INFO:journalctl@ceph.osd.7.smithi194.stdout:Mar 20 08:48:44 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-osd-7[44872]: 2024-03-20T08:48:44.630+0000 7f37d5961700 -1 osd.7 71 heartbeat_check: no reply from 172.21.15.183:6806 osd.0 since back 2024-03-20T08:48:16.746885+0000 front 2024-03-20T08:48:16.746994+0000 (oldest deadline 2024-03-20T08:48:42.046867+0000) 2024-03-20T08:48:45.454 INFO:journalctl@ceph.osd.3.smithi183.stdout:Mar 20 08:48:45 smithi183 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-osd-3[47180]: 2024-03-20T08:48:45.059+0000 7f5430e9d700 -1 osd.3 71 heartbeat_check: no reply from 172.21.15.183:6806 osd.0 since back 2024-03-20T08:48:15.142331+0000 front 2024-03-20T08:48:15.142245+0000 (oldest deadline 2024-03-20T08:48:41.042229+0000) 2024-03-20T08:48:45.590 INFO:journalctl@ceph.osd.4.smithi194.stdout:Mar 20 08:48:45 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-osd-4[34266]: 2024-03-20T08:48:45.252+0000 7fd5262d1700 -1 osd.4 71 heartbeat_check: no reply from 172.21.15.183:6806 osd.0 since back 2024-03-20T08:48:16.599224+0000 front 2024-03-20T08:48:16.599321+0000 (oldest deadline 2024-03-20T08:48:40.699108+0000) 2024-03-20T08:48:45.590 INFO:journalctl@ceph.osd.6.smithi194.stdout:Mar 20 08:48:45 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-osd-6[41285]: 2024-03-20T08:48:45.445+0000 7fabb4d64700 -1 osd.6 71 heartbeat_check: no reply from 172.21.15.183:6806 osd.0 since back 2024-03-20T08:48:18.168836+0000 front 2024-03-20T08:48:18.168757+0000 (oldest deadline 2024-03-20T08:48:40.468676+0000) 2024-03-20T08:48:45.889 INFO:journalctl@ceph.osd.5.smithi194.stdout:Mar 20 08:48:45 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-osd-5[37760]: 2024-03-20T08:48:45.590+0000 7f60920aa700 -1 osd.5 71 heartbeat_check: no reply from 172.21.15.183:6806 osd.0 since back 2024-03-20T08:48:15.356781+0000 front 2024-03-20T08:48:15.356583+0000 (oldest deadline 2024-03-20T08:48:40.656479+0000) 2024-03-20T08:48:45.889 INFO:journalctl@ceph.osd.7.smithi194.stdout:Mar 20 08:48:45 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-osd-7[44872]: 2024-03-20T08:48:45.649+0000 7f37d5961700 -1 osd.7 71 heartbeat_check: no reply from 172.21.15.183:6806 osd.0 since back 2024-03-20T08:48:16.746885+0000 front 2024-03-20T08:48:16.746994+0000 (oldest deadline 2024-03-20T08:48:42.046867+0000) 2024-03-20T08:48:46.596 INFO:journalctl@ceph.osd.3.smithi183.stdout:Mar 20 08:48:46 smithi183 podman[73851]: 2024-03-20 08:48:46.313606576 +0000 UTC m=+6.107005305 container died 77e843f75ce441a23c7e04c968b0c1ea95a7a89db69f948c1e5f0cf1001a4c30 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:976a8b2d06e3c6bfd12488297e8584dd704cc06c3e976c7e0f718c13e238e912, name=ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-osd-3, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.vendor=CentOS, RELEASE=wip-yuri5-testing-2024-03-18-1144-e7e80a6, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5) 2024-03-20T08:48:46.602 INFO:journalctl@ceph.osd.4.smithi194.stdout:Mar 20 08:48:46 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-osd-4[34266]: 2024-03-20T08:48:46.228+0000 7fd5262d1700 -1 osd.4 71 heartbeat_check: no reply from 172.21.15.183:6806 osd.0 since back 2024-03-20T08:48:16.599224+0000 front 2024-03-20T08:48:16.599321+0000 (oldest deadline 2024-03-20T08:48:40.699108+0000) 2024-03-20T08:48:46.602 INFO:journalctl@ceph.osd.6.smithi194.stdout:Mar 20 08:48:46 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-osd-6[41285]: 2024-03-20T08:48:46.427+0000 7fabb4d64700 -1 osd.6 71 heartbeat_check: no reply from 172.21.15.183:6806 osd.0 since back 2024-03-20T08:48:18.168836+0000 front 2024-03-20T08:48:18.168757+0000 (oldest deadline 2024-03-20T08:48:40.468676+0000) 2024-03-20T08:48:46.889 INFO:journalctl@ceph.osd.5.smithi194.stdout:Mar 20 08:48:46 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-osd-5[37760]: 2024-03-20T08:48:46.602+0000 7f60920aa700 -1 osd.5 71 heartbeat_check: no reply from 172.21.15.183:6806 osd.0 since back 2024-03-20T08:48:15.356781+0000 front 2024-03-20T08:48:15.356583+0000 (oldest deadline 2024-03-20T08:48:40.656479+0000) 2024-03-20T08:48:46.889 INFO:journalctl@ceph.osd.7.smithi194.stdout:Mar 20 08:48:46 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-osd-7[44872]: 2024-03-20T08:48:46.673+0000 7f37d5961700 -1 osd.7 71 heartbeat_check: no reply from 172.21.15.183:6806 osd.0 since back 2024-03-20T08:48:16.746885+0000 front 2024-03-20T08:48:16.746994+0000 (oldest deadline 2024-03-20T08:48:42.046867+0000) 2024-03-20T08:48:46.946 INFO:journalctl@ceph.osd.3.smithi183.stdout:Mar 20 08:48:46 smithi183 podman[73851]: 2024-03-20 08:48:46.761793386 +0000 UTC m=+6.555192147 container cleanup 77e843f75ce441a23c7e04c968b0c1ea95a7a89db69f948c1e5f0cf1001a4c30 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:976a8b2d06e3c6bfd12488297e8584dd704cc06c3e976c7e0f718c13e238e912, name=ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-osd-3, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, RELEASE=wip-yuri5-testing-2024-03-18-1144-e7e80a6, GIT_CLEAN=True, maintainer=Guillaume Abrioux ) 2024-03-20T08:48:46.946 INFO:journalctl@ceph.osd.3.smithi183.stdout:Mar 20 08:48:46 smithi183 bash[73851]: ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-osd-3 2024-03-20T08:48:47.204 INFO:journalctl@ceph.osd.3.smithi183.stdout:Mar 20 08:48:46 smithi183 podman[74316]: 2024-03-20 08:48:46.946325401 +0000 UTC m=+0.627405189 container remove 77e843f75ce441a23c7e04c968b0c1ea95a7a89db69f948c1e5f0cf1001a4c30 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:976a8b2d06e3c6bfd12488297e8584dd704cc06c3e976c7e0f718c13e238e912, name=ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-osd-3, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-yuri5-testing-2024-03-18-1144-e7e80a6, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.build-date=20240307, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD) 2024-03-20T08:48:47.585 INFO:journalctl@ceph.osd.4.smithi194.stdout:Mar 20 08:48:47 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-osd-4[34266]: 2024-03-20T08:48:47.224+0000 7fd5262d1700 -1 osd.4 71 heartbeat_check: no reply from 172.21.15.183:6806 osd.0 since back 2024-03-20T08:48:16.599224+0000 front 2024-03-20T08:48:16.599321+0000 (oldest deadline 2024-03-20T08:48:40.699108+0000) 2024-03-20T08:48:47.586 INFO:journalctl@ceph.osd.6.smithi194.stdout:Mar 20 08:48:47 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-osd-6[41285]: 2024-03-20T08:48:47.426+0000 7fabb4d64700 -1 osd.6 71 heartbeat_check: no reply from 172.21.15.183:6806 osd.0 since back 2024-03-20T08:48:18.168836+0000 front 2024-03-20T08:48:18.168757+0000 (oldest deadline 2024-03-20T08:48:40.468676+0000) 2024-03-20T08:48:47.624 INFO:journalctl@ceph.osd.3.smithi183.stdout:Mar 20 08:48:47 smithi183 podman[74409]: 2024-03-20 08:48:47.255781372 +0000 UTC m=+0.018276271 image pull 9ceba81fb2c8d8682cb159527017d591637b153b3aa4a390ffa8586db6dcda00 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:976a8b2d06e3c6bfd12488297e8584dd704cc06c3e976c7e0f718c13e238e912 2024-03-20T08:48:47.624 INFO:journalctl@ceph.osd.3.smithi183.stdout:Mar 20 08:48:47 smithi183 podman[74409]: 2024-03-20 08:48:47.355441084 +0000 UTC m=+0.117935956 container create 3e3d96a47a73fdc0f937bc75fceeda9990794f8b7c50f443db9419090f1082c8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:976a8b2d06e3c6bfd12488297e8584dd704cc06c3e976c7e0f718c13e238e912, name=ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-osd-3-deactivate, ceph=True, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240307, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-yuri5-testing-2024-03-18-1144-e7e80a6, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-20T08:48:47.624 INFO:journalctl@ceph.osd.3.smithi183.stdout:Mar 20 08:48:47 smithi183 podman[74409]: 2024-03-20 08:48:47.549321097 +0000 UTC m=+0.311815968 container init 3e3d96a47a73fdc0f937bc75fceeda9990794f8b7c50f443db9419090f1082c8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:976a8b2d06e3c6bfd12488297e8584dd704cc06c3e976c7e0f718c13e238e912, name=ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-osd-3-deactivate, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.build-date=20240307, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, RELEASE=wip-yuri5-testing-2024-03-18-1144-e7e80a6) 2024-03-20T08:48:47.624 INFO:journalctl@ceph.osd.3.smithi183.stdout:Mar 20 08:48:47 smithi183 podman[74409]: 2024-03-20 08:48:47.552323353 +0000 UTC m=+0.314818226 container start 3e3d96a47a73fdc0f937bc75fceeda9990794f8b7c50f443db9419090f1082c8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:976a8b2d06e3c6bfd12488297e8584dd704cc06c3e976c7e0f718c13e238e912, name=ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-osd-3-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240307, GIT_CLEAN=True, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, ceph=True, RELEASE=wip-yuri5-testing-2024-03-18-1144-e7e80a6, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-20T08:48:47.624 INFO:journalctl@ceph.osd.3.smithi183.stdout:Mar 20 08:48:47 smithi183 podman[74409]: 2024-03-20 08:48:47.624138395 +0000 UTC m=+0.386633276 container attach 3e3d96a47a73fdc0f937bc75fceeda9990794f8b7c50f443db9419090f1082c8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:976a8b2d06e3c6bfd12488297e8584dd704cc06c3e976c7e0f718c13e238e912, name=ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-osd-3-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , GIT_CLEAN=True, RELEASE=wip-yuri5-testing-2024-03-18-1144-e7e80a6, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240307, org.label-schema.license=GPLv2, ceph=True) 2024-03-20T08:48:47.889 INFO:journalctl@ceph.osd.5.smithi194.stdout:Mar 20 08:48:47 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-osd-5[37760]: 2024-03-20T08:48:47.585+0000 7f60920aa700 -1 osd.5 71 heartbeat_check: no reply from 172.21.15.183:6806 osd.0 since back 2024-03-20T08:48:15.356781+0000 front 2024-03-20T08:48:15.356583+0000 (oldest deadline 2024-03-20T08:48:40.656479+0000) 2024-03-20T08:48:47.889 INFO:journalctl@ceph.osd.7.smithi194.stdout:Mar 20 08:48:47 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-osd-7[44872]: 2024-03-20T08:48:47.645+0000 7f37d5961700 -1 osd.7 71 heartbeat_check: no reply from 172.21.15.183:6806 osd.0 since back 2024-03-20T08:48:16.746885+0000 front 2024-03-20T08:48:16.746994+0000 (oldest deadline 2024-03-20T08:48:42.046867+0000) 2024-03-20T08:48:47.954 INFO:journalctl@ceph.osd.3.smithi183.stdout:Mar 20 08:48:47 smithi183 podman[74409]: 2024-03-20 08:48:47.848622462 +0000 UTC m=+0.611117333 container died 3e3d96a47a73fdc0f937bc75fceeda9990794f8b7c50f443db9419090f1082c8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:976a8b2d06e3c6bfd12488297e8584dd704cc06c3e976c7e0f718c13e238e912, name=ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-osd-3-deactivate, org.label-schema.build-date=20240307, GIT_BRANCH=HEAD, GIT_CLEAN=True, RELEASE=wip-yuri5-testing-2024-03-18-1144-e7e80a6, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, ceph=True, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2) 2024-03-20T08:48:48.481 DEBUG:teuthology.orchestra.run.smithi183:> sudo pkill -f 'journalctl -f -n 0 -u ceph-4c3f3eba-e695-11ee-95c9-87774f69a715@osd.3.service' 2024-03-20T08:48:48.522 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-20T08:48:48.522 INFO:tasks.cephadm.osd.3:Stopped osd.3 2024-03-20T08:48:48.522 INFO:tasks.cephadm.osd.4:Stopping osd.4... 2024-03-20T08:48:48.523 DEBUG:teuthology.orchestra.run.smithi194:> sudo systemctl stop ceph-4c3f3eba-e695-11ee-95c9-87774f69a715@osd.4 2024-03-20T08:48:48.537 INFO:journalctl@ceph.osd.4.smithi194.stdout:Mar 20 08:48:48 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-osd-4[34266]: 2024-03-20T08:48:48.183+0000 7fd5262d1700 -1 osd.4 71 heartbeat_check: no reply from 172.21.15.183:6806 osd.0 since back 2024-03-20T08:48:16.599224+0000 front 2024-03-20T08:48:16.599321+0000 (oldest deadline 2024-03-20T08:48:40.699108+0000) 2024-03-20T08:48:48.537 INFO:journalctl@ceph.osd.6.smithi194.stdout:Mar 20 08:48:48 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-osd-6[41285]: 2024-03-20T08:48:48.378+0000 7fabb4d64700 -1 osd.6 71 heartbeat_check: no reply from 172.21.15.183:6806 osd.0 since back 2024-03-20T08:48:18.168836+0000 front 2024-03-20T08:48:18.168757+0000 (oldest deadline 2024-03-20T08:48:40.468676+0000) 2024-03-20T08:48:48.818 INFO:journalctl@ceph.osd.4.smithi194.stdout:Mar 20 08:48:48 smithi194 systemd[1]: Stopping Ceph osd.4 for 4c3f3eba-e695-11ee-95c9-87774f69a715... 2024-03-20T08:48:48.819 INFO:journalctl@ceph.osd.5.smithi194.stdout:Mar 20 08:48:48 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-osd-5[37760]: 2024-03-20T08:48:48.536+0000 7f60920aa700 -1 osd.5 71 heartbeat_check: no reply from 172.21.15.183:6806 osd.0 since back 2024-03-20T08:48:15.356781+0000 front 2024-03-20T08:48:15.356583+0000 (oldest deadline 2024-03-20T08:48:40.656479+0000) 2024-03-20T08:48:48.819 INFO:journalctl@ceph.osd.7.smithi194.stdout:Mar 20 08:48:48 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-osd-7[44872]: 2024-03-20T08:48:48.667+0000 7f37d5961700 -1 osd.7 71 heartbeat_check: no reply from 172.21.15.183:6806 osd.0 since back 2024-03-20T08:48:16.746885+0000 front 2024-03-20T08:48:16.746994+0000 (oldest deadline 2024-03-20T08:48:42.046867+0000) 2024-03-20T08:48:49.138 INFO:journalctl@ceph.osd.4.smithi194.stdout:Mar 20 08:48:48 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-osd-4[34266]: 2024-03-20T08:48:48.818+0000 7fd529cdd700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.4 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-20T08:48:49.139 INFO:journalctl@ceph.osd.4.smithi194.stdout:Mar 20 08:48:48 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-osd-4[34266]: 2024-03-20T08:48:48.818+0000 7fd529cdd700 -1 osd.4 71 *** Got signal Terminated *** 2024-03-20T08:48:49.139 INFO:journalctl@ceph.osd.4.smithi194.stdout:Mar 20 08:48:48 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-osd-4[34266]: 2024-03-20T08:48:48.818+0000 7fd529cdd700 -1 osd.4 71 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-20T08:48:49.421 INFO:journalctl@ceph.osd.4.smithi194.stdout:Mar 20 08:48:49 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-osd-4[34266]: 2024-03-20T08:48:49.160+0000 7fd5262d1700 -1 osd.4 71 heartbeat_check: no reply from 172.21.15.183:6806 osd.0 since back 2024-03-20T08:48:16.599224+0000 front 2024-03-20T08:48:16.599321+0000 (oldest deadline 2024-03-20T08:48:40.699108+0000) 2024-03-20T08:48:49.693 INFO:journalctl@ceph.osd.5.smithi194.stdout:Mar 20 08:48:49 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-osd-5[37760]: 2024-03-20T08:48:49.535+0000 7f60920aa700 -1 osd.5 71 heartbeat_check: no reply from 172.21.15.183:6806 osd.0 since back 2024-03-20T08:48:15.356781+0000 front 2024-03-20T08:48:15.356583+0000 (oldest deadline 2024-03-20T08:48:40.656479+0000) 2024-03-20T08:48:49.694 INFO:journalctl@ceph.osd.6.smithi194.stdout:Mar 20 08:48:49 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-osd-6[41285]: 2024-03-20T08:48:49.421+0000 7fabb4d64700 -1 osd.6 71 heartbeat_check: no reply from 172.21.15.183:6806 osd.0 since back 2024-03-20T08:48:18.168836+0000 front 2024-03-20T08:48:18.168757+0000 (oldest deadline 2024-03-20T08:48:40.468676+0000) 2024-03-20T08:48:49.694 INFO:journalctl@ceph.osd.6.smithi194.stdout:Mar 20 08:48:49 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-osd-6[41285]: 2024-03-20T08:48:49.421+0000 7fabb4d64700 -1 osd.6 71 heartbeat_check: no reply from 172.21.15.183:6814 osd.1 since back 2024-03-20T08:48:26.269930+0000 front 2024-03-20T08:48:26.269946+0000 (oldest deadline 2024-03-20T08:48:49.169783+0000) 2024-03-20T08:48:50.128 INFO:journalctl@ceph.osd.7.smithi194.stdout:Mar 20 08:48:49 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-osd-7[44872]: 2024-03-20T08:48:49.693+0000 7f37d5961700 -1 osd.7 71 heartbeat_check: no reply from 172.21.15.183:6806 osd.0 since back 2024-03-20T08:48:16.746885+0000 front 2024-03-20T08:48:16.746994+0000 (oldest deadline 2024-03-20T08:48:42.046867+0000) 2024-03-20T08:48:50.389 INFO:journalctl@ceph.osd.4.smithi194.stdout:Mar 20 08:48:50 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-osd-4[34266]: 2024-03-20T08:48:50.127+0000 7fd5262d1700 -1 osd.4 71 heartbeat_check: no reply from 172.21.15.183:6806 osd.0 since back 2024-03-20T08:48:16.599224+0000 front 2024-03-20T08:48:16.599321+0000 (oldest deadline 2024-03-20T08:48:40.699108+0000) 2024-03-20T08:48:50.389 INFO:journalctl@ceph.osd.4.smithi194.stdout:Mar 20 08:48:50 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-osd-4[34266]: 2024-03-20T08:48:50.127+0000 7fd5262d1700 -1 osd.4 71 heartbeat_check: no reply from 172.21.15.183:6814 osd.1 since back 2024-03-20T08:48:26.000114+0000 front 2024-03-20T08:48:26.000181+0000 (oldest deadline 2024-03-20T08:48:49.500019+0000) 2024-03-20T08:48:50.680 INFO:journalctl@ceph.osd.5.smithi194.stdout:Mar 20 08:48:50 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-osd-5[37760]: 2024-03-20T08:48:50.515+0000 7f60920aa700 -1 osd.5 71 heartbeat_check: no reply from 172.21.15.183:6806 osd.0 since back 2024-03-20T08:48:15.356781+0000 front 2024-03-20T08:48:15.356583+0000 (oldest deadline 2024-03-20T08:48:40.656479+0000) 2024-03-20T08:48:50.681 INFO:journalctl@ceph.osd.6.smithi194.stdout:Mar 20 08:48:50 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-osd-6[41285]: 2024-03-20T08:48:50.393+0000 7fabb4d64700 -1 osd.6 71 heartbeat_check: no reply from 172.21.15.183:6806 osd.0 since back 2024-03-20T08:48:18.168836+0000 front 2024-03-20T08:48:18.168757+0000 (oldest deadline 2024-03-20T08:48:40.468676+0000) 2024-03-20T08:48:50.681 INFO:journalctl@ceph.osd.6.smithi194.stdout:Mar 20 08:48:50 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-osd-6[41285]: 2024-03-20T08:48:50.393+0000 7fabb4d64700 -1 osd.6 71 heartbeat_check: no reply from 172.21.15.183:6814 osd.1 since back 2024-03-20T08:48:26.269930+0000 front 2024-03-20T08:48:26.269946+0000 (oldest deadline 2024-03-20T08:48:49.169783+0000) 2024-03-20T08:48:51.138 INFO:journalctl@ceph.osd.7.smithi194.stdout:Mar 20 08:48:50 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-osd-7[44872]: 2024-03-20T08:48:50.680+0000 7f37d5961700 -1 osd.7 71 heartbeat_check: no reply from 172.21.15.183:6806 osd.0 since back 2024-03-20T08:48:16.746885+0000 front 2024-03-20T08:48:16.746994+0000 (oldest deadline 2024-03-20T08:48:42.046867+0000) 2024-03-20T08:48:51.494 INFO:journalctl@ceph.osd.6.smithi194.stdout:Mar 20 08:48:51 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-osd-6[41285]: 2024-03-20T08:48:51.346+0000 7fabb4d64700 -1 osd.6 71 heartbeat_check: no reply from 172.21.15.183:6806 osd.0 since back 2024-03-20T08:48:18.168836+0000 front 2024-03-20T08:48:18.168757+0000 (oldest deadline 2024-03-20T08:48:40.468676+0000) 2024-03-20T08:48:51.494 INFO:journalctl@ceph.osd.6.smithi194.stdout:Mar 20 08:48:51 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-osd-6[41285]: 2024-03-20T08:48:51.346+0000 7fabb4d64700 -1 osd.6 71 heartbeat_check: no reply from 172.21.15.183:6814 osd.1 since back 2024-03-20T08:48:26.269930+0000 front 2024-03-20T08:48:26.269946+0000 (oldest deadline 2024-03-20T08:48:49.169783+0000) 2024-03-20T08:48:51.495 INFO:journalctl@ceph.osd.4.smithi194.stdout:Mar 20 08:48:51 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-osd-4[34266]: 2024-03-20T08:48:51.176+0000 7fd5262d1700 -1 osd.4 71 heartbeat_check: no reply from 172.21.15.183:6806 osd.0 since back 2024-03-20T08:48:16.599224+0000 front 2024-03-20T08:48:16.599321+0000 (oldest deadline 2024-03-20T08:48:40.699108+0000) 2024-03-20T08:48:51.495 INFO:journalctl@ceph.osd.4.smithi194.stdout:Mar 20 08:48:51 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-osd-4[34266]: 2024-03-20T08:48:51.176+0000 7fd5262d1700 -1 osd.4 71 heartbeat_check: no reply from 172.21.15.183:6814 osd.1 since back 2024-03-20T08:48:26.000114+0000 front 2024-03-20T08:48:26.000181+0000 (oldest deadline 2024-03-20T08:48:49.500019+0000) 2024-03-20T08:48:51.889 INFO:journalctl@ceph.osd.5.smithi194.stdout:Mar 20 08:48:51 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-osd-5[37760]: 2024-03-20T08:48:51.494+0000 7f60920aa700 -1 osd.5 71 heartbeat_check: no reply from 172.21.15.183:6806 osd.0 since back 2024-03-20T08:48:15.356781+0000 front 2024-03-20T08:48:15.356583+0000 (oldest deadline 2024-03-20T08:48:40.656479+0000) 2024-03-20T08:48:51.889 INFO:journalctl@ceph.osd.7.smithi194.stdout:Mar 20 08:48:51 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-osd-7[44872]: 2024-03-20T08:48:51.697+0000 7f37d5961700 -1 osd.7 71 heartbeat_check: no reply from 172.21.15.183:6806 osd.0 since back 2024-03-20T08:48:16.746885+0000 front 2024-03-20T08:48:16.746994+0000 (oldest deadline 2024-03-20T08:48:42.046867+0000) 2024-03-20T08:48:51.889 INFO:journalctl@ceph.osd.7.smithi194.stdout:Mar 20 08:48:51 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-osd-7[44872]: 2024-03-20T08:48:51.697+0000 7f37d5961700 -1 osd.7 71 heartbeat_check: no reply from 172.21.15.183:6814 osd.1 since back 2024-03-20T08:48:26.147608+0000 front 2024-03-20T08:48:26.147671+0000 (oldest deadline 2024-03-20T08:48:50.847640+0000) 2024-03-20T08:48:52.515 INFO:journalctl@ceph.osd.4.smithi194.stdout:Mar 20 08:48:52 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-osd-4[34266]: 2024-03-20T08:48:52.171+0000 7fd5262d1700 -1 osd.4 71 heartbeat_check: no reply from 172.21.15.183:6806 osd.0 since back 2024-03-20T08:48:16.599224+0000 front 2024-03-20T08:48:16.599321+0000 (oldest deadline 2024-03-20T08:48:40.699108+0000) 2024-03-20T08:48:52.515 INFO:journalctl@ceph.osd.4.smithi194.stdout:Mar 20 08:48:52 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-osd-4[34266]: 2024-03-20T08:48:52.171+0000 7fd5262d1700 -1 osd.4 71 heartbeat_check: no reply from 172.21.15.183:6814 osd.1 since back 2024-03-20T08:48:26.000114+0000 front 2024-03-20T08:48:26.000181+0000 (oldest deadline 2024-03-20T08:48:49.500019+0000) 2024-03-20T08:48:52.515 INFO:journalctl@ceph.osd.6.smithi194.stdout:Mar 20 08:48:52 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-osd-6[41285]: 2024-03-20T08:48:52.311+0000 7fabb4d64700 -1 osd.6 71 heartbeat_check: no reply from 172.21.15.183:6806 osd.0 since back 2024-03-20T08:48:18.168836+0000 front 2024-03-20T08:48:18.168757+0000 (oldest deadline 2024-03-20T08:48:40.468676+0000) 2024-03-20T08:48:52.515 INFO:journalctl@ceph.osd.6.smithi194.stdout:Mar 20 08:48:52 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-osd-6[41285]: 2024-03-20T08:48:52.311+0000 7fabb4d64700 -1 osd.6 71 heartbeat_check: no reply from 172.21.15.183:6814 osd.1 since back 2024-03-20T08:48:26.269930+0000 front 2024-03-20T08:48:26.269946+0000 (oldest deadline 2024-03-20T08:48:49.169783+0000) 2024-03-20T08:48:52.887 INFO:journalctl@ceph.osd.7.smithi194.stdout:Mar 20 08:48:52 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-osd-7[44872]: 2024-03-20T08:48:52.713+0000 7f37d5961700 -1 osd.7 71 heartbeat_check: no reply from 172.21.15.183:6806 osd.0 since back 2024-03-20T08:48:16.746885+0000 front 2024-03-20T08:48:16.746994+0000 (oldest deadline 2024-03-20T08:48:42.046867+0000) 2024-03-20T08:48:52.887 INFO:journalctl@ceph.osd.7.smithi194.stdout:Mar 20 08:48:52 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-osd-7[44872]: 2024-03-20T08:48:52.713+0000 7f37d5961700 -1 osd.7 71 heartbeat_check: no reply from 172.21.15.183:6814 osd.1 since back 2024-03-20T08:48:26.147608+0000 front 2024-03-20T08:48:26.147671+0000 (oldest deadline 2024-03-20T08:48:50.847640+0000) 2024-03-20T08:48:52.887 INFO:journalctl@ceph.osd.5.smithi194.stdout:Mar 20 08:48:52 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-osd-5[37760]: 2024-03-20T08:48:52.514+0000 7f60920aa700 -1 osd.5 71 heartbeat_check: no reply from 172.21.15.183:6806 osd.0 since back 2024-03-20T08:48:15.356781+0000 front 2024-03-20T08:48:15.356583+0000 (oldest deadline 2024-03-20T08:48:40.656479+0000) 2024-03-20T08:48:52.887 INFO:journalctl@ceph.osd.5.smithi194.stdout:Mar 20 08:48:52 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-osd-5[37760]: 2024-03-20T08:48:52.514+0000 7f60920aa700 -1 osd.5 71 heartbeat_check: no reply from 172.21.15.183:6814 osd.1 since back 2024-03-20T08:48:26.358498+0000 front 2024-03-20T08:48:26.358394+0000 (oldest deadline 2024-03-20T08:48:51.658281+0000) 2024-03-20T08:48:53.449 INFO:journalctl@ceph.osd.6.smithi194.stdout:Mar 20 08:48:53 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-osd-6[41285]: 2024-03-20T08:48:53.311+0000 7fabb4d64700 -1 osd.6 71 heartbeat_check: no reply from 172.21.15.183:6806 osd.0 since back 2024-03-20T08:48:18.168836+0000 front 2024-03-20T08:48:18.168757+0000 (oldest deadline 2024-03-20T08:48:40.468676+0000) 2024-03-20T08:48:53.449 INFO:journalctl@ceph.osd.6.smithi194.stdout:Mar 20 08:48:53 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-osd-6[41285]: 2024-03-20T08:48:53.311+0000 7fabb4d64700 -1 osd.6 71 heartbeat_check: no reply from 172.21.15.183:6814 osd.1 since back 2024-03-20T08:48:26.269930+0000 front 2024-03-20T08:48:26.269946+0000 (oldest deadline 2024-03-20T08:48:49.169783+0000) 2024-03-20T08:48:53.449 INFO:journalctl@ceph.osd.4.smithi194.stdout:Mar 20 08:48:53 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-osd-4[34266]: 2024-03-20T08:48:53.173+0000 7fd5262d1700 -1 osd.4 71 heartbeat_check: no reply from 172.21.15.183:6806 osd.0 since back 2024-03-20T08:48:16.599224+0000 front 2024-03-20T08:48:16.599321+0000 (oldest deadline 2024-03-20T08:48:40.699108+0000) 2024-03-20T08:48:53.449 INFO:journalctl@ceph.osd.4.smithi194.stdout:Mar 20 08:48:53 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-osd-4[34266]: 2024-03-20T08:48:53.173+0000 7fd5262d1700 -1 osd.4 71 heartbeat_check: no reply from 172.21.15.183:6814 osd.1 since back 2024-03-20T08:48:26.000114+0000 front 2024-03-20T08:48:26.000181+0000 (oldest deadline 2024-03-20T08:48:49.500019+0000) 2024-03-20T08:48:53.739 INFO:journalctl@ceph.osd.5.smithi194.stdout:Mar 20 08:48:53 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-osd-5[37760]: 2024-03-20T08:48:53.561+0000 7f60920aa700 -1 osd.5 71 heartbeat_check: no reply from 172.21.15.183:6806 osd.0 since back 2024-03-20T08:48:15.356781+0000 front 2024-03-20T08:48:15.356583+0000 (oldest deadline 2024-03-20T08:48:40.656479+0000) 2024-03-20T08:48:53.739 INFO:journalctl@ceph.osd.5.smithi194.stdout:Mar 20 08:48:53 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-osd-5[37760]: 2024-03-20T08:48:53.561+0000 7f60920aa700 -1 osd.5 71 heartbeat_check: no reply from 172.21.15.183:6814 osd.1 since back 2024-03-20T08:48:26.358498+0000 front 2024-03-20T08:48:26.358394+0000 (oldest deadline 2024-03-20T08:48:51.658281+0000) 2024-03-20T08:48:54.138 INFO:journalctl@ceph.osd.7.smithi194.stdout:Mar 20 08:48:53 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-osd-7[44872]: 2024-03-20T08:48:53.738+0000 7f37d5961700 -1 osd.7 71 heartbeat_check: no reply from 172.21.15.183:6806 osd.0 since back 2024-03-20T08:48:16.746885+0000 front 2024-03-20T08:48:16.746994+0000 (oldest deadline 2024-03-20T08:48:42.046867+0000) 2024-03-20T08:48:54.139 INFO:journalctl@ceph.osd.7.smithi194.stdout:Mar 20 08:48:53 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-osd-7[44872]: 2024-03-20T08:48:53.738+0000 7f37d5961700 -1 osd.7 71 heartbeat_check: no reply from 172.21.15.183:6814 osd.1 since back 2024-03-20T08:48:26.147608+0000 front 2024-03-20T08:48:26.147671+0000 (oldest deadline 2024-03-20T08:48:50.847640+0000) 2024-03-20T08:48:54.403 INFO:journalctl@ceph.osd.4.smithi194.stdout:Mar 20 08:48:54 smithi194 podman[62635]: 2024-03-20 08:48:54.143428829 +0000 UTC m=+5.424231117 container died aac22f466d474a6ead1914ecf91e07c054bee8d3db0085bf384ca8db6531e86d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:976a8b2d06e3c6bfd12488297e8584dd704cc06c3e976c7e0f718c13e238e912, name=ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-osd-4, org.label-schema.build-date=20240307, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, ceph=True, io.buildah.version=1.33.5, RELEASE=wip-yuri5-testing-2024-03-18-1144-e7e80a6, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2) 2024-03-20T08:48:54.403 INFO:journalctl@ceph.osd.6.smithi194.stdout:Mar 20 08:48:54 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-osd-6[41285]: 2024-03-20T08:48:54.316+0000 7fabb4d64700 -1 osd.6 71 heartbeat_check: no reply from 172.21.15.183:6806 osd.0 since back 2024-03-20T08:48:18.168836+0000 front 2024-03-20T08:48:18.168757+0000 (oldest deadline 2024-03-20T08:48:40.468676+0000) 2024-03-20T08:48:54.403 INFO:journalctl@ceph.osd.6.smithi194.stdout:Mar 20 08:48:54 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-osd-6[41285]: 2024-03-20T08:48:54.316+0000 7fabb4d64700 -1 osd.6 71 heartbeat_check: no reply from 172.21.15.183:6814 osd.1 since back 2024-03-20T08:48:26.269930+0000 front 2024-03-20T08:48:26.269946+0000 (oldest deadline 2024-03-20T08:48:49.169783+0000) 2024-03-20T08:48:54.725 INFO:journalctl@ceph.osd.4.smithi194.stdout:Mar 20 08:48:54 smithi194 podman[62635]: 2024-03-20 08:48:54.543694464 +0000 UTC m=+5.824496793 container cleanup aac22f466d474a6ead1914ecf91e07c054bee8d3db0085bf384ca8db6531e86d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:976a8b2d06e3c6bfd12488297e8584dd704cc06c3e976c7e0f718c13e238e912, name=ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-osd-4, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, RELEASE=wip-yuri5-testing-2024-03-18-1144-e7e80a6, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240307, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5) 2024-03-20T08:48:54.725 INFO:journalctl@ceph.osd.4.smithi194.stdout:Mar 20 08:48:54 smithi194 bash[62635]: ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-osd-4 2024-03-20T08:48:54.725 INFO:journalctl@ceph.osd.5.smithi194.stdout:Mar 20 08:48:54 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-osd-5[37760]: 2024-03-20T08:48:54.556+0000 7f60920aa700 -1 osd.5 71 heartbeat_check: no reply from 172.21.15.183:6806 osd.0 since back 2024-03-20T08:48:15.356781+0000 front 2024-03-20T08:48:15.356583+0000 (oldest deadline 2024-03-20T08:48:40.656479+0000) 2024-03-20T08:48:54.725 INFO:journalctl@ceph.osd.5.smithi194.stdout:Mar 20 08:48:54 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-osd-5[37760]: 2024-03-20T08:48:54.556+0000 7f60920aa700 -1 osd.5 71 heartbeat_check: no reply from 172.21.15.183:6814 osd.1 since back 2024-03-20T08:48:26.358498+0000 front 2024-03-20T08:48:26.358394+0000 (oldest deadline 2024-03-20T08:48:51.658281+0000) 2024-03-20T08:48:55.118 INFO:journalctl@ceph.osd.4.smithi194.stdout:Mar 20 08:48:54 smithi194 podman[62821]: 2024-03-20 08:48:54.78688639 +0000 UTC m=+0.638166564 container remove aac22f466d474a6ead1914ecf91e07c054bee8d3db0085bf384ca8db6531e86d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:976a8b2d06e3c6bfd12488297e8584dd704cc06c3e976c7e0f718c13e238e912, name=ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-osd-4, ceph=True, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.build-date=20240307, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-yuri5-testing-2024-03-18-1144-e7e80a6, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2) 2024-03-20T08:48:55.119 INFO:journalctl@ceph.osd.7.smithi194.stdout:Mar 20 08:48:54 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-osd-7[44872]: 2024-03-20T08:48:54.724+0000 7f37d5961700 -1 osd.7 71 heartbeat_check: no reply from 172.21.15.183:6806 osd.0 since back 2024-03-20T08:48:16.746885+0000 front 2024-03-20T08:48:16.746994+0000 (oldest deadline 2024-03-20T08:48:42.046867+0000) 2024-03-20T08:48:55.119 INFO:journalctl@ceph.osd.7.smithi194.stdout:Mar 20 08:48:54 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-osd-7[44872]: 2024-03-20T08:48:54.724+0000 7f37d5961700 -1 osd.7 71 heartbeat_check: no reply from 172.21.15.183:6814 osd.1 since back 2024-03-20T08:48:26.147608+0000 front 2024-03-20T08:48:26.147671+0000 (oldest deadline 2024-03-20T08:48:50.847640+0000) 2024-03-20T08:48:55.389 INFO:journalctl@ceph.osd.6.smithi194.stdout:Mar 20 08:48:55 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-osd-6[41285]: 2024-03-20T08:48:55.334+0000 7fabb4d64700 -1 osd.6 71 heartbeat_check: no reply from 172.21.15.183:6806 osd.0 since back 2024-03-20T08:48:18.168836+0000 front 2024-03-20T08:48:18.168757+0000 (oldest deadline 2024-03-20T08:48:40.468676+0000) 2024-03-20T08:48:55.389 INFO:journalctl@ceph.osd.6.smithi194.stdout:Mar 20 08:48:55 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-osd-6[41285]: 2024-03-20T08:48:55.334+0000 7fabb4d64700 -1 osd.6 71 heartbeat_check: no reply from 172.21.15.183:6814 osd.1 since back 2024-03-20T08:48:26.269930+0000 front 2024-03-20T08:48:26.269946+0000 (oldest deadline 2024-03-20T08:48:49.169783+0000) 2024-03-20T08:48:55.389 INFO:journalctl@ceph.osd.4.smithi194.stdout:Mar 20 08:48:55 smithi194 podman[62946]: 2024-03-20 08:48:55.023405438 +0000 UTC m=+0.020276206 image pull 9ceba81fb2c8d8682cb159527017d591637b153b3aa4a390ffa8586db6dcda00 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:976a8b2d06e3c6bfd12488297e8584dd704cc06c3e976c7e0f718c13e238e912 2024-03-20T08:48:55.389 INFO:journalctl@ceph.osd.4.smithi194.stdout:Mar 20 08:48:55 smithi194 podman[62946]: 2024-03-20 08:48:55.138048137 +0000 UTC m=+0.134918921 container create 978ffffb711ec265ad9084f16dfbc0ef3f4b298700aa34c325b979d71127507e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:976a8b2d06e3c6bfd12488297e8584dd704cc06c3e976c7e0f718c13e238e912, name=ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-osd-4-deactivate, io.buildah.version=1.33.5, ceph=True, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, RELEASE=wip-yuri5-testing-2024-03-18-1144-e7e80a6) 2024-03-20T08:48:55.389 INFO:journalctl@ceph.osd.4.smithi194.stdout:Mar 20 08:48:55 smithi194 podman[62946]: 2024-03-20 08:48:55.356645408 +0000 UTC m=+0.353516145 container init 978ffffb711ec265ad9084f16dfbc0ef3f4b298700aa34c325b979d71127507e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:976a8b2d06e3c6bfd12488297e8584dd704cc06c3e976c7e0f718c13e238e912, name=ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-osd-4-deactivate, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-yuri5-testing-2024-03-18-1144-e7e80a6, org.label-schema.license=GPLv2, ceph=True, GIT_CLEAN=True, org.label-schema.build-date=20240307, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-20T08:48:55.389 INFO:journalctl@ceph.osd.4.smithi194.stdout:Mar 20 08:48:55 smithi194 podman[62946]: 2024-03-20 08:48:55.359929821 +0000 UTC m=+0.356800567 container start 978ffffb711ec265ad9084f16dfbc0ef3f4b298700aa34c325b979d71127507e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:976a8b2d06e3c6bfd12488297e8584dd704cc06c3e976c7e0f718c13e238e912, name=ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-osd-4-deactivate, GIT_BRANCH=HEAD, GIT_CLEAN=True, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, ceph=True, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.build-date=20240307, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-yuri5-testing-2024-03-18-1144-e7e80a6, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-20T08:48:55.693 INFO:journalctl@ceph.osd.4.smithi194.stdout:Mar 20 08:48:55 smithi194 podman[62946]: 2024-03-20 08:48:55.457092116 +0000 UTC m=+0.453962852 container attach 978ffffb711ec265ad9084f16dfbc0ef3f4b298700aa34c325b979d71127507e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:976a8b2d06e3c6bfd12488297e8584dd704cc06c3e976c7e0f718c13e238e912, name=ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-osd-4-deactivate, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.build-date=20240307, ceph=True, GIT_CLEAN=True, maintainer=Guillaume Abrioux , RELEASE=wip-yuri5-testing-2024-03-18-1144-e7e80a6, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-20T08:48:55.693 INFO:journalctl@ceph.osd.5.smithi194.stdout:Mar 20 08:48:55 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-osd-5[37760]: 2024-03-20T08:48:55.506+0000 7f60920aa700 -1 osd.5 71 heartbeat_check: no reply from 172.21.15.183:6806 osd.0 since back 2024-03-20T08:48:15.356781+0000 front 2024-03-20T08:48:15.356583+0000 (oldest deadline 2024-03-20T08:48:40.656479+0000) 2024-03-20T08:48:55.694 INFO:journalctl@ceph.osd.5.smithi194.stdout:Mar 20 08:48:55 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-osd-5[37760]: 2024-03-20T08:48:55.506+0000 7f60920aa700 -1 osd.5 71 heartbeat_check: no reply from 172.21.15.183:6814 osd.1 since back 2024-03-20T08:48:26.358498+0000 front 2024-03-20T08:48:26.358394+0000 (oldest deadline 2024-03-20T08:48:51.658281+0000) 2024-03-20T08:48:55.985 INFO:journalctl@ceph.osd.4.smithi194.stdout:Mar 20 08:48:55 smithi194 podman[62946]: 2024-03-20 08:48:55.693714668 +0000 UTC m=+0.690585400 container died 978ffffb711ec265ad9084f16dfbc0ef3f4b298700aa34c325b979d71127507e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:976a8b2d06e3c6bfd12488297e8584dd704cc06c3e976c7e0f718c13e238e912, name=ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-osd-4-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, RELEASE=wip-yuri5-testing-2024-03-18-1144-e7e80a6, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240307, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, GIT_CLEAN=True, CEPH_POINT_RELEASE=) 2024-03-20T08:48:55.985 INFO:journalctl@ceph.osd.7.smithi194.stdout:Mar 20 08:48:55 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-osd-7[44872]: 2024-03-20T08:48:55.700+0000 7f37d5961700 -1 osd.7 71 heartbeat_check: no reply from 172.21.15.183:6806 osd.0 since back 2024-03-20T08:48:16.746885+0000 front 2024-03-20T08:48:16.746994+0000 (oldest deadline 2024-03-20T08:48:42.046867+0000) 2024-03-20T08:48:55.985 INFO:journalctl@ceph.osd.7.smithi194.stdout:Mar 20 08:48:55 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-osd-7[44872]: 2024-03-20T08:48:55.700+0000 7f37d5961700 -1 osd.7 71 heartbeat_check: no reply from 172.21.15.183:6814 osd.1 since back 2024-03-20T08:48:26.147608+0000 front 2024-03-20T08:48:26.147671+0000 (oldest deadline 2024-03-20T08:48:50.847640+0000) 2024-03-20T08:48:56.302 DEBUG:teuthology.orchestra.run.smithi194:> sudo pkill -f 'journalctl -f -n 0 -u ceph-4c3f3eba-e695-11ee-95c9-87774f69a715@osd.4.service' 2024-03-20T08:48:56.390 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-20T08:48:56.390 INFO:tasks.cephadm.osd.4:Stopped osd.4 2024-03-20T08:48:56.390 INFO:tasks.cephadm.osd.5:Stopping osd.5... 2024-03-20T08:48:56.390 DEBUG:teuthology.orchestra.run.smithi194:> sudo systemctl stop ceph-4c3f3eba-e695-11ee-95c9-87774f69a715@osd.5 2024-03-20T08:48:56.535 INFO:journalctl@ceph.osd.6.smithi194.stdout:Mar 20 08:48:56 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-osd-6[41285]: 2024-03-20T08:48:56.357+0000 7fabb4d64700 -1 osd.6 71 heartbeat_check: no reply from 172.21.15.183:6806 osd.0 since back 2024-03-20T08:48:18.168836+0000 front 2024-03-20T08:48:18.168757+0000 (oldest deadline 2024-03-20T08:48:40.468676+0000) 2024-03-20T08:48:56.535 INFO:journalctl@ceph.osd.6.smithi194.stdout:Mar 20 08:48:56 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-osd-6[41285]: 2024-03-20T08:48:56.357+0000 7fabb4d64700 -1 osd.6 71 heartbeat_check: no reply from 172.21.15.183:6814 osd.1 since back 2024-03-20T08:48:26.269930+0000 front 2024-03-20T08:48:26.269946+0000 (oldest deadline 2024-03-20T08:48:49.169783+0000) 2024-03-20T08:48:56.536 INFO:journalctl@ceph.osd.5.smithi194.stdout:Mar 20 08:48:56 smithi194 systemd[1]: Stopping Ceph osd.5 for 4c3f3eba-e695-11ee-95c9-87774f69a715... 2024-03-20T08:48:56.536 INFO:journalctl@ceph.osd.5.smithi194.stdout:Mar 20 08:48:56 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-osd-5[37760]: 2024-03-20T08:48:56.471+0000 7f60920aa700 -1 osd.5 71 heartbeat_check: no reply from 172.21.15.183:6806 osd.0 since back 2024-03-20T08:48:15.356781+0000 front 2024-03-20T08:48:15.356583+0000 (oldest deadline 2024-03-20T08:48:40.656479+0000) 2024-03-20T08:48:56.536 INFO:journalctl@ceph.osd.5.smithi194.stdout:Mar 20 08:48:56 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-osd-5[37760]: 2024-03-20T08:48:56.471+0000 7f60920aa700 -1 osd.5 71 heartbeat_check: no reply from 172.21.15.183:6814 osd.1 since back 2024-03-20T08:48:26.358498+0000 front 2024-03-20T08:48:26.358394+0000 (oldest deadline 2024-03-20T08:48:51.658281+0000) 2024-03-20T08:48:56.888 INFO:journalctl@ceph.osd.7.smithi194.stdout:Mar 20 08:48:56 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-osd-7[44872]: 2024-03-20T08:48:56.705+0000 7f37d5961700 -1 osd.7 71 heartbeat_check: no reply from 172.21.15.183:6806 osd.0 since back 2024-03-20T08:48:16.746885+0000 front 2024-03-20T08:48:16.746994+0000 (oldest deadline 2024-03-20T08:48:42.046867+0000) 2024-03-20T08:48:56.889 INFO:journalctl@ceph.osd.7.smithi194.stdout:Mar 20 08:48:56 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-osd-7[44872]: 2024-03-20T08:48:56.705+0000 7f37d5961700 -1 osd.7 71 heartbeat_check: no reply from 172.21.15.183:6814 osd.1 since back 2024-03-20T08:48:26.147608+0000 front 2024-03-20T08:48:26.147671+0000 (oldest deadline 2024-03-20T08:48:50.847640+0000) 2024-03-20T08:48:57.415 INFO:journalctl@ceph.osd.5.smithi194.stdout:Mar 20 08:48:57 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-osd-5[37760]: 2024-03-20T08:48:57.140+0000 7f6099abe700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.5 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-20T08:48:57.415 INFO:journalctl@ceph.osd.5.smithi194.stdout:Mar 20 08:48:57 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-osd-5[37760]: 2024-03-20T08:48:57.140+0000 7f6099abe700 -1 osd.5 71 *** Got signal Terminated *** 2024-03-20T08:48:57.415 INFO:journalctl@ceph.osd.5.smithi194.stdout:Mar 20 08:48:57 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-osd-5[37760]: 2024-03-20T08:48:57.140+0000 7f6099abe700 -1 osd.5 71 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-20T08:48:57.415 INFO:journalctl@ceph.osd.6.smithi194.stdout:Mar 20 08:48:57 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-osd-6[41285]: 2024-03-20T08:48:57.308+0000 7fabb4d64700 -1 osd.6 71 heartbeat_check: no reply from 172.21.15.183:6806 osd.0 since back 2024-03-20T08:48:18.168836+0000 front 2024-03-20T08:48:18.168757+0000 (oldest deadline 2024-03-20T08:48:40.468676+0000) 2024-03-20T08:48:57.415 INFO:journalctl@ceph.osd.6.smithi194.stdout:Mar 20 08:48:57 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-osd-6[41285]: 2024-03-20T08:48:57.308+0000 7fabb4d64700 -1 osd.6 71 heartbeat_check: no reply from 172.21.15.183:6814 osd.1 since back 2024-03-20T08:48:26.269930+0000 front 2024-03-20T08:48:26.269946+0000 (oldest deadline 2024-03-20T08:48:49.169783+0000) 2024-03-20T08:48:57.749 INFO:journalctl@ceph.osd.5.smithi194.stdout:Mar 20 08:48:57 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-osd-5[37760]: 2024-03-20T08:48:57.456+0000 7f60920aa700 -1 osd.5 71 heartbeat_check: no reply from 172.21.15.183:6806 osd.0 since back 2024-03-20T08:48:15.356781+0000 front 2024-03-20T08:48:15.356583+0000 (oldest deadline 2024-03-20T08:48:40.656479+0000) 2024-03-20T08:48:57.750 INFO:journalctl@ceph.osd.5.smithi194.stdout:Mar 20 08:48:57 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-osd-5[37760]: 2024-03-20T08:48:57.456+0000 7f60920aa700 -1 osd.5 71 heartbeat_check: no reply from 172.21.15.183:6814 osd.1 since back 2024-03-20T08:48:26.358498+0000 front 2024-03-20T08:48:26.358394+0000 (oldest deadline 2024-03-20T08:48:51.658281+0000) 2024-03-20T08:48:58.118 INFO:journalctl@ceph.osd.7.smithi194.stdout:Mar 20 08:48:57 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-osd-7[44872]: 2024-03-20T08:48:57.749+0000 7f37d5961700 -1 osd.7 71 heartbeat_check: no reply from 172.21.15.183:6806 osd.0 since back 2024-03-20T08:48:16.746885+0000 front 2024-03-20T08:48:16.746994+0000 (oldest deadline 2024-03-20T08:48:42.046867+0000) 2024-03-20T08:48:58.118 INFO:journalctl@ceph.osd.7.smithi194.stdout:Mar 20 08:48:57 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-osd-7[44872]: 2024-03-20T08:48:57.749+0000 7f37d5961700 -1 osd.7 71 heartbeat_check: no reply from 172.21.15.183:6814 osd.1 since back 2024-03-20T08:48:26.147608+0000 front 2024-03-20T08:48:26.147671+0000 (oldest deadline 2024-03-20T08:48:50.847640+0000) 2024-03-20T08:48:58.369 INFO:journalctl@ceph.osd.6.smithi194.stdout:Mar 20 08:48:58 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-osd-6[41285]: 2024-03-20T08:48:58.275+0000 7fabb4d64700 -1 osd.6 71 heartbeat_check: no reply from 172.21.15.183:6806 osd.0 since back 2024-03-20T08:48:18.168836+0000 front 2024-03-20T08:48:18.168757+0000 (oldest deadline 2024-03-20T08:48:40.468676+0000) 2024-03-20T08:48:58.369 INFO:journalctl@ceph.osd.6.smithi194.stdout:Mar 20 08:48:58 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-osd-6[41285]: 2024-03-20T08:48:58.275+0000 7fabb4d64700 -1 osd.6 71 heartbeat_check: no reply from 172.21.15.183:6814 osd.1 since back 2024-03-20T08:48:26.269930+0000 front 2024-03-20T08:48:26.269946+0000 (oldest deadline 2024-03-20T08:48:49.169783+0000) 2024-03-20T08:48:58.639 INFO:journalctl@ceph.osd.5.smithi194.stdout:Mar 20 08:48:58 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-osd-5[37760]: 2024-03-20T08:48:58.440+0000 7f60920aa700 -1 osd.5 71 heartbeat_check: no reply from 172.21.15.183:6806 osd.0 since back 2024-03-20T08:48:15.356781+0000 front 2024-03-20T08:48:15.356583+0000 (oldest deadline 2024-03-20T08:48:40.656479+0000) 2024-03-20T08:48:58.639 INFO:journalctl@ceph.osd.5.smithi194.stdout:Mar 20 08:48:58 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-osd-5[37760]: 2024-03-20T08:48:58.440+0000 7f60920aa700 -1 osd.5 71 heartbeat_check: no reply from 172.21.15.183:6814 osd.1 since back 2024-03-20T08:48:26.358498+0000 front 2024-03-20T08:48:26.358394+0000 (oldest deadline 2024-03-20T08:48:51.658281+0000) 2024-03-20T08:48:58.639 INFO:journalctl@ceph.osd.5.smithi194.stdout:Mar 20 08:48:58 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-osd-5[37760]: 2024-03-20T08:48:58.440+0000 7f60920aa700 -1 osd.5 71 heartbeat_check: no reply from 172.21.15.183:6822 osd.2 since back 2024-03-20T08:48:35.759253+0000 front 2024-03-20T08:48:35.759287+0000 (oldest deadline 2024-03-20T08:48:58.059267+0000) 2024-03-20T08:48:59.056 INFO:journalctl@ceph.osd.7.smithi194.stdout:Mar 20 08:48:58 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-osd-7[44872]: 2024-03-20T08:48:58.774+0000 7f37d5961700 -1 osd.7 71 heartbeat_check: no reply from 172.21.15.183:6806 osd.0 since back 2024-03-20T08:48:16.746885+0000 front 2024-03-20T08:48:16.746994+0000 (oldest deadline 2024-03-20T08:48:42.046867+0000) 2024-03-20T08:48:59.057 INFO:journalctl@ceph.osd.7.smithi194.stdout:Mar 20 08:48:58 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-osd-7[44872]: 2024-03-20T08:48:58.774+0000 7f37d5961700 -1 osd.7 71 heartbeat_check: no reply from 172.21.15.183:6814 osd.1 since back 2024-03-20T08:48:26.147608+0000 front 2024-03-20T08:48:26.147671+0000 (oldest deadline 2024-03-20T08:48:50.847640+0000) 2024-03-20T08:48:59.389 INFO:journalctl@ceph.osd.6.smithi194.stdout:Mar 20 08:48:59 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-osd-6[41285]: 2024-03-20T08:48:59.302+0000 7fabb4d64700 -1 osd.6 71 heartbeat_check: no reply from 172.21.15.183:6806 osd.0 since back 2024-03-20T08:48:18.168836+0000 front 2024-03-20T08:48:18.168757+0000 (oldest deadline 2024-03-20T08:48:40.468676+0000) 2024-03-20T08:48:59.389 INFO:journalctl@ceph.osd.6.smithi194.stdout:Mar 20 08:48:59 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-osd-6[41285]: 2024-03-20T08:48:59.302+0000 7fabb4d64700 -1 osd.6 71 heartbeat_check: no reply from 172.21.15.183:6814 osd.1 since back 2024-03-20T08:48:26.269930+0000 front 2024-03-20T08:48:26.269946+0000 (oldest deadline 2024-03-20T08:48:49.169783+0000) 2024-03-20T08:48:59.802 INFO:journalctl@ceph.osd.5.smithi194.stdout:Mar 20 08:48:59 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-osd-5[37760]: 2024-03-20T08:48:59.487+0000 7f60920aa700 -1 osd.5 71 heartbeat_check: no reply from 172.21.15.183:6806 osd.0 since back 2024-03-20T08:48:15.356781+0000 front 2024-03-20T08:48:15.356583+0000 (oldest deadline 2024-03-20T08:48:40.656479+0000) 2024-03-20T08:48:59.802 INFO:journalctl@ceph.osd.5.smithi194.stdout:Mar 20 08:48:59 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-osd-5[37760]: 2024-03-20T08:48:59.487+0000 7f60920aa700 -1 osd.5 71 heartbeat_check: no reply from 172.21.15.183:6814 osd.1 since back 2024-03-20T08:48:26.358498+0000 front 2024-03-20T08:48:26.358394+0000 (oldest deadline 2024-03-20T08:48:51.658281+0000) 2024-03-20T08:48:59.802 INFO:journalctl@ceph.osd.5.smithi194.stdout:Mar 20 08:48:59 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-osd-5[37760]: 2024-03-20T08:48:59.487+0000 7f60920aa700 -1 osd.5 71 heartbeat_check: no reply from 172.21.15.183:6822 osd.2 since back 2024-03-20T08:48:35.759253+0000 front 2024-03-20T08:48:35.759287+0000 (oldest deadline 2024-03-20T08:48:58.059267+0000) 2024-03-20T08:49:00.079 INFO:journalctl@ceph.osd.7.smithi194.stdout:Mar 20 08:48:59 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-osd-7[44872]: 2024-03-20T08:48:59.803+0000 7f37d5961700 -1 osd.7 71 heartbeat_check: no reply from 172.21.15.183:6806 osd.0 since back 2024-03-20T08:48:16.746885+0000 front 2024-03-20T08:48:16.746994+0000 (oldest deadline 2024-03-20T08:48:42.046867+0000) 2024-03-20T08:49:00.079 INFO:journalctl@ceph.osd.7.smithi194.stdout:Mar 20 08:48:59 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-osd-7[44872]: 2024-03-20T08:48:59.803+0000 7f37d5961700 -1 osd.7 71 heartbeat_check: no reply from 172.21.15.183:6814 osd.1 since back 2024-03-20T08:48:26.147608+0000 front 2024-03-20T08:48:26.147671+0000 (oldest deadline 2024-03-20T08:48:50.847640+0000) 2024-03-20T08:49:00.079 INFO:journalctl@ceph.osd.7.smithi194.stdout:Mar 20 08:48:59 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-osd-7[44872]: 2024-03-20T08:48:59.803+0000 7f37d5961700 -1 osd.7 71 heartbeat_check: no reply from 172.21.15.183:6822 osd.2 since back 2024-03-20T08:48:36.148515+0000 front 2024-03-20T08:48:36.148487+0000 (oldest deadline 2024-03-20T08:48:59.048455+0000) 2024-03-20T08:49:00.388 INFO:journalctl@ceph.osd.6.smithi194.stdout:Mar 20 08:49:00 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-osd-6[41285]: 2024-03-20T08:49:00.321+0000 7fabb4d64700 -1 osd.6 71 heartbeat_check: no reply from 172.21.15.183:6806 osd.0 since back 2024-03-20T08:48:18.168836+0000 front 2024-03-20T08:48:18.168757+0000 (oldest deadline 2024-03-20T08:48:40.468676+0000) 2024-03-20T08:49:00.389 INFO:journalctl@ceph.osd.6.smithi194.stdout:Mar 20 08:49:00 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-osd-6[41285]: 2024-03-20T08:49:00.321+0000 7fabb4d64700 -1 osd.6 71 heartbeat_check: no reply from 172.21.15.183:6814 osd.1 since back 2024-03-20T08:48:26.269930+0000 front 2024-03-20T08:48:26.269946+0000 (oldest deadline 2024-03-20T08:48:49.169783+0000) 2024-03-20T08:49:00.858 INFO:journalctl@ceph.osd.5.smithi194.stdout:Mar 20 08:49:00 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-osd-5[37760]: 2024-03-20T08:49:00.533+0000 7f60920aa700 -1 osd.5 71 heartbeat_check: no reply from 172.21.15.183:6806 osd.0 since back 2024-03-20T08:48:15.356781+0000 front 2024-03-20T08:48:15.356583+0000 (oldest deadline 2024-03-20T08:48:40.656479+0000) 2024-03-20T08:49:00.859 INFO:journalctl@ceph.osd.5.smithi194.stdout:Mar 20 08:49:00 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-osd-5[37760]: 2024-03-20T08:49:00.533+0000 7f60920aa700 -1 osd.5 71 heartbeat_check: no reply from 172.21.15.183:6814 osd.1 since back 2024-03-20T08:48:26.358498+0000 front 2024-03-20T08:48:26.358394+0000 (oldest deadline 2024-03-20T08:48:51.658281+0000) 2024-03-20T08:49:00.859 INFO:journalctl@ceph.osd.5.smithi194.stdout:Mar 20 08:49:00 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-osd-5[37760]: 2024-03-20T08:49:00.533+0000 7f60920aa700 -1 osd.5 71 heartbeat_check: no reply from 172.21.15.183:6822 osd.2 since back 2024-03-20T08:48:35.759253+0000 front 2024-03-20T08:48:35.759287+0000 (oldest deadline 2024-03-20T08:48:58.059267+0000) 2024-03-20T08:49:00.859 INFO:journalctl@ceph.osd.7.smithi194.stdout:Mar 20 08:49:00 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-osd-7[44872]: 2024-03-20T08:49:00.761+0000 7f37d5961700 -1 osd.7 71 heartbeat_check: no reply from 172.21.15.183:6806 osd.0 since back 2024-03-20T08:48:16.746885+0000 front 2024-03-20T08:48:16.746994+0000 (oldest deadline 2024-03-20T08:48:42.046867+0000) 2024-03-20T08:49:00.859 INFO:journalctl@ceph.osd.7.smithi194.stdout:Mar 20 08:49:00 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-osd-7[44872]: 2024-03-20T08:49:00.761+0000 7f37d5961700 -1 osd.7 71 heartbeat_check: no reply from 172.21.15.183:6814 osd.1 since back 2024-03-20T08:48:26.147608+0000 front 2024-03-20T08:48:26.147671+0000 (oldest deadline 2024-03-20T08:48:50.847640+0000) 2024-03-20T08:49:00.859 INFO:journalctl@ceph.osd.7.smithi194.stdout:Mar 20 08:49:00 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-osd-7[44872]: 2024-03-20T08:49:00.761+0000 7f37d5961700 -1 osd.7 71 heartbeat_check: no reply from 172.21.15.183:6822 osd.2 since back 2024-03-20T08:48:36.148515+0000 front 2024-03-20T08:48:36.148487+0000 (oldest deadline 2024-03-20T08:48:59.048455+0000) 2024-03-20T08:49:00.859 INFO:journalctl@ceph.osd.7.smithi194.stdout:Mar 20 08:49:00 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-osd-7[44872]: 2024-03-20T08:49:00.761+0000 7f37d5961700 -1 osd.7 71 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.14556.0:149 2.0 2:353ec8f9:::gateway.conf:head [getxattr epoch in=5b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e71) 2024-03-20T08:49:01.526 INFO:journalctl@ceph.osd.6.smithi194.stdout:Mar 20 08:49:01 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-osd-6[41285]: 2024-03-20T08:49:01.341+0000 7fabb4d64700 -1 osd.6 71 heartbeat_check: no reply from 172.21.15.183:6806 osd.0 since back 2024-03-20T08:48:18.168836+0000 front 2024-03-20T08:48:18.168757+0000 (oldest deadline 2024-03-20T08:48:40.468676+0000) 2024-03-20T08:49:01.526 INFO:journalctl@ceph.osd.6.smithi194.stdout:Mar 20 08:49:01 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-osd-6[41285]: 2024-03-20T08:49:01.341+0000 7fabb4d64700 -1 osd.6 71 heartbeat_check: no reply from 172.21.15.183:6814 osd.1 since back 2024-03-20T08:48:26.269930+0000 front 2024-03-20T08:48:26.269946+0000 (oldest deadline 2024-03-20T08:48:49.169783+0000) 2024-03-20T08:49:01.889 INFO:journalctl@ceph.osd.5.smithi194.stdout:Mar 20 08:49:01 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-osd-5[37760]: 2024-03-20T08:49:01.525+0000 7f60920aa700 -1 osd.5 71 heartbeat_check: no reply from 172.21.15.183:6806 osd.0 since back 2024-03-20T08:48:15.356781+0000 front 2024-03-20T08:48:15.356583+0000 (oldest deadline 2024-03-20T08:48:40.656479+0000) 2024-03-20T08:49:01.889 INFO:journalctl@ceph.osd.5.smithi194.stdout:Mar 20 08:49:01 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-osd-5[37760]: 2024-03-20T08:49:01.525+0000 7f60920aa700 -1 osd.5 71 heartbeat_check: no reply from 172.21.15.183:6814 osd.1 since back 2024-03-20T08:48:26.358498+0000 front 2024-03-20T08:48:26.358394+0000 (oldest deadline 2024-03-20T08:48:51.658281+0000) 2024-03-20T08:49:01.889 INFO:journalctl@ceph.osd.5.smithi194.stdout:Mar 20 08:49:01 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-osd-5[37760]: 2024-03-20T08:49:01.525+0000 7f60920aa700 -1 osd.5 71 heartbeat_check: no reply from 172.21.15.183:6822 osd.2 since back 2024-03-20T08:48:35.759253+0000 front 2024-03-20T08:48:35.759287+0000 (oldest deadline 2024-03-20T08:48:58.059267+0000) 2024-03-20T08:49:01.889 INFO:journalctl@ceph.osd.7.smithi194.stdout:Mar 20 08:49:01 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-osd-7[44872]: 2024-03-20T08:49:01.758+0000 7f37d5961700 -1 osd.7 71 heartbeat_check: no reply from 172.21.15.183:6806 osd.0 since back 2024-03-20T08:48:16.746885+0000 front 2024-03-20T08:48:16.746994+0000 (oldest deadline 2024-03-20T08:48:42.046867+0000) 2024-03-20T08:49:01.890 INFO:journalctl@ceph.osd.7.smithi194.stdout:Mar 20 08:49:01 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-osd-7[44872]: 2024-03-20T08:49:01.758+0000 7f37d5961700 -1 osd.7 71 heartbeat_check: no reply from 172.21.15.183:6814 osd.1 since back 2024-03-20T08:48:26.147608+0000 front 2024-03-20T08:48:26.147671+0000 (oldest deadline 2024-03-20T08:48:50.847640+0000) 2024-03-20T08:49:01.890 INFO:journalctl@ceph.osd.7.smithi194.stdout:Mar 20 08:49:01 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-osd-7[44872]: 2024-03-20T08:49:01.758+0000 7f37d5961700 -1 osd.7 71 heartbeat_check: no reply from 172.21.15.183:6822 osd.2 since back 2024-03-20T08:48:36.148515+0000 front 2024-03-20T08:48:36.148487+0000 (oldest deadline 2024-03-20T08:48:59.048455+0000) 2024-03-20T08:49:01.890 INFO:journalctl@ceph.osd.7.smithi194.stdout:Mar 20 08:49:01 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-osd-7[44872]: 2024-03-20T08:49:01.759+0000 7f37d5961700 -1 osd.7 71 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.14556.0:149 2.0 2:353ec8f9:::gateway.conf:head [getxattr epoch in=5b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e71) 2024-03-20T08:49:02.639 INFO:journalctl@ceph.osd.6.smithi194.stdout:Mar 20 08:49:02 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-osd-6[41285]: 2024-03-20T08:49:02.374+0000 7fabb4d64700 -1 osd.6 71 heartbeat_check: no reply from 172.21.15.183:6806 osd.0 since back 2024-03-20T08:48:18.168836+0000 front 2024-03-20T08:48:18.168757+0000 (oldest deadline 2024-03-20T08:48:40.468676+0000) 2024-03-20T08:49:02.639 INFO:journalctl@ceph.osd.6.smithi194.stdout:Mar 20 08:49:02 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-osd-6[41285]: 2024-03-20T08:49:02.374+0000 7fabb4d64700 -1 osd.6 71 heartbeat_check: no reply from 172.21.15.183:6814 osd.1 since back 2024-03-20T08:48:26.269930+0000 front 2024-03-20T08:48:26.269946+0000 (oldest deadline 2024-03-20T08:48:49.169783+0000) 2024-03-20T08:49:02.639 INFO:journalctl@ceph.osd.6.smithi194.stdout:Mar 20 08:49:02 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-osd-6[41285]: 2024-03-20T08:49:02.374+0000 7fabb4d64700 -1 osd.6 71 heartbeat_check: no reply from 172.21.15.183:6822 osd.2 since back 2024-03-20T08:48:35.571240+0000 front 2024-03-20T08:48:35.571309+0000 (oldest deadline 2024-03-20T08:49:01.471211+0000) 2024-03-20T08:49:02.954 INFO:journalctl@ceph.osd.5.smithi194.stdout:Mar 20 08:49:02 smithi194 podman[63193]: 2024-03-20 08:49:02.681639746 +0000 UTC m=+5.664583782 container died 234c30f1c954108f5dbe7b2e2241ee3889d18738d1c29fece33268a4e7c6a1f9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:976a8b2d06e3c6bfd12488297e8584dd704cc06c3e976c7e0f718c13e238e912, name=ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-osd-5, CEPH_POINT_RELEASE=, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.build-date=20240307, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-yuri5-testing-2024-03-18-1144-e7e80a6, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, ceph=True) 2024-03-20T08:49:02.955 INFO:journalctl@ceph.osd.7.smithi194.stdout:Mar 20 08:49:02 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-osd-7[44872]: 2024-03-20T08:49:02.773+0000 7f37d5961700 -1 osd.7 71 heartbeat_check: no reply from 172.21.15.183:6806 osd.0 since back 2024-03-20T08:48:16.746885+0000 front 2024-03-20T08:48:16.746994+0000 (oldest deadline 2024-03-20T08:48:42.046867+0000) 2024-03-20T08:49:02.955 INFO:journalctl@ceph.osd.7.smithi194.stdout:Mar 20 08:49:02 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-osd-7[44872]: 2024-03-20T08:49:02.773+0000 7f37d5961700 -1 osd.7 71 heartbeat_check: no reply from 172.21.15.183:6814 osd.1 since back 2024-03-20T08:48:26.147608+0000 front 2024-03-20T08:48:26.147671+0000 (oldest deadline 2024-03-20T08:48:50.847640+0000) 2024-03-20T08:49:02.955 INFO:journalctl@ceph.osd.7.smithi194.stdout:Mar 20 08:49:02 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-osd-7[44872]: 2024-03-20T08:49:02.773+0000 7f37d5961700 -1 osd.7 71 heartbeat_check: no reply from 172.21.15.183:6822 osd.2 since back 2024-03-20T08:48:36.148515+0000 front 2024-03-20T08:48:36.148487+0000 (oldest deadline 2024-03-20T08:48:59.048455+0000) 2024-03-20T08:49:02.955 INFO:journalctl@ceph.osd.7.smithi194.stdout:Mar 20 08:49:02 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-osd-7[44872]: 2024-03-20T08:49:02.774+0000 7f37d5961700 -1 osd.7 71 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.14556.0:149 2.0 2:353ec8f9:::gateway.conf:head [getxattr epoch in=5b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e71) 2024-03-20T08:49:03.330 INFO:journalctl@ceph.osd.5.smithi194.stdout:Mar 20 08:49:03 smithi194 podman[63193]: 2024-03-20 08:49:03.054363905 +0000 UTC m=+6.037307933 container cleanup 234c30f1c954108f5dbe7b2e2241ee3889d18738d1c29fece33268a4e7c6a1f9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:976a8b2d06e3c6bfd12488297e8584dd704cc06c3e976c7e0f718c13e238e912, name=ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-osd-5, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240307, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, ceph=True, io.buildah.version=1.33.5, RELEASE=wip-yuri5-testing-2024-03-18-1144-e7e80a6, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True) 2024-03-20T08:49:03.330 INFO:journalctl@ceph.osd.5.smithi194.stdout:Mar 20 08:49:03 smithi194 bash[63193]: ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-osd-5 2024-03-20T08:49:03.639 INFO:journalctl@ceph.osd.5.smithi194.stdout:Mar 20 08:49:03 smithi194 podman[63674]: 2024-03-20 08:49:03.330531629 +0000 UTC m=+0.649381000 container remove 234c30f1c954108f5dbe7b2e2241ee3889d18738d1c29fece33268a4e7c6a1f9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:976a8b2d06e3c6bfd12488297e8584dd704cc06c3e976c7e0f718c13e238e912, name=ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-osd-5, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-yuri5-testing-2024-03-18-1144-e7e80a6, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, org.label-schema.build-date=20240307, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_CLEAN=True) 2024-03-20T08:49:03.639 INFO:journalctl@ceph.osd.6.smithi194.stdout:Mar 20 08:49:03 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-osd-6[41285]: 2024-03-20T08:49:03.383+0000 7fabb4d64700 -1 osd.6 71 heartbeat_check: no reply from 172.21.15.183:6806 osd.0 since back 2024-03-20T08:48:18.168836+0000 front 2024-03-20T08:48:18.168757+0000 (oldest deadline 2024-03-20T08:48:40.468676+0000) 2024-03-20T08:49:03.639 INFO:journalctl@ceph.osd.6.smithi194.stdout:Mar 20 08:49:03 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-osd-6[41285]: 2024-03-20T08:49:03.383+0000 7fabb4d64700 -1 osd.6 71 heartbeat_check: no reply from 172.21.15.183:6814 osd.1 since back 2024-03-20T08:48:26.269930+0000 front 2024-03-20T08:48:26.269946+0000 (oldest deadline 2024-03-20T08:48:49.169783+0000) 2024-03-20T08:49:03.639 INFO:journalctl@ceph.osd.6.smithi194.stdout:Mar 20 08:49:03 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-osd-6[41285]: 2024-03-20T08:49:03.383+0000 7fabb4d64700 -1 osd.6 71 heartbeat_check: no reply from 172.21.15.183:6822 osd.2 since back 2024-03-20T08:48:35.571240+0000 front 2024-03-20T08:48:35.571309+0000 (oldest deadline 2024-03-20T08:49:01.471211+0000) 2024-03-20T08:49:03.983 INFO:journalctl@ceph.osd.5.smithi194.stdout:Mar 20 08:49:03 smithi194 podman[63767]: 2024-03-20 08:49:03.566506988 +0000 UTC m=+0.018759211 image pull 9ceba81fb2c8d8682cb159527017d591637b153b3aa4a390ffa8586db6dcda00 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:976a8b2d06e3c6bfd12488297e8584dd704cc06c3e976c7e0f718c13e238e912 2024-03-20T08:49:03.984 INFO:journalctl@ceph.osd.5.smithi194.stdout:Mar 20 08:49:03 smithi194 podman[63767]: 2024-03-20 08:49:03.673117504 +0000 UTC m=+0.125369709 container create 57d4132d927700dd02ae09c3d84b0dbb7306f0db26a69dbcd89b56b90f9de502 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:976a8b2d06e3c6bfd12488297e8584dd704cc06c3e976c7e0f718c13e238e912, name=ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-osd-5-deactivate, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-yuri5-testing-2024-03-18-1144-e7e80a6, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, ceph=True) 2024-03-20T08:49:03.984 INFO:journalctl@ceph.osd.5.smithi194.stdout:Mar 20 08:49:03 smithi194 podman[63767]: 2024-03-20 08:49:03.900362241 +0000 UTC m=+0.352614458 container init 57d4132d927700dd02ae09c3d84b0dbb7306f0db26a69dbcd89b56b90f9de502 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:976a8b2d06e3c6bfd12488297e8584dd704cc06c3e976c7e0f718c13e238e912, name=ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-osd-5-deactivate, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, io.buildah.version=1.33.5, org.label-schema.build-date=20240307, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-yuri5-testing-2024-03-18-1144-e7e80a6, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0) 2024-03-20T08:49:03.984 INFO:journalctl@ceph.osd.5.smithi194.stdout:Mar 20 08:49:03 smithi194 podman[63767]: 2024-03-20 08:49:03.903604862 +0000 UTC m=+0.355857071 container start 57d4132d927700dd02ae09c3d84b0dbb7306f0db26a69dbcd89b56b90f9de502 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:976a8b2d06e3c6bfd12488297e8584dd704cc06c3e976c7e0f718c13e238e912, name=ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-osd-5-deactivate, CEPH_POINT_RELEASE=, RELEASE=wip-yuri5-testing-2024-03-18-1144-e7e80a6, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, ceph=True, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.build-date=20240307) 2024-03-20T08:49:03.984 INFO:journalctl@ceph.osd.7.smithi194.stdout:Mar 20 08:49:03 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-osd-7[44872]: 2024-03-20T08:49:03.802+0000 7f37d5961700 -1 osd.7 71 heartbeat_check: no reply from 172.21.15.183:6806 osd.0 since back 2024-03-20T08:48:16.746885+0000 front 2024-03-20T08:48:16.746994+0000 (oldest deadline 2024-03-20T08:48:42.046867+0000) 2024-03-20T08:49:03.984 INFO:journalctl@ceph.osd.7.smithi194.stdout:Mar 20 08:49:03 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-osd-7[44872]: 2024-03-20T08:49:03.802+0000 7f37d5961700 -1 osd.7 71 heartbeat_check: no reply from 172.21.15.183:6814 osd.1 since back 2024-03-20T08:48:26.147608+0000 front 2024-03-20T08:48:26.147671+0000 (oldest deadline 2024-03-20T08:48:50.847640+0000) 2024-03-20T08:49:03.984 INFO:journalctl@ceph.osd.7.smithi194.stdout:Mar 20 08:49:03 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-osd-7[44872]: 2024-03-20T08:49:03.802+0000 7f37d5961700 -1 osd.7 71 heartbeat_check: no reply from 172.21.15.183:6822 osd.2 since back 2024-03-20T08:48:36.148515+0000 front 2024-03-20T08:48:36.148487+0000 (oldest deadline 2024-03-20T08:48:59.048455+0000) 2024-03-20T08:49:03.984 INFO:journalctl@ceph.osd.7.smithi194.stdout:Mar 20 08:49:03 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-osd-7[44872]: 2024-03-20T08:49:03.802+0000 7f37d5961700 -1 osd.7 71 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.14556.0:149 2.0 2:353ec8f9:::gateway.conf:head [getxattr epoch in=5b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e71) 2024-03-20T08:49:04.342 INFO:journalctl@ceph.osd.5.smithi194.stdout:Mar 20 08:49:03 smithi194 podman[63767]: 2024-03-20 08:49:03.983575734 +0000 UTC m=+0.435827956 container attach 57d4132d927700dd02ae09c3d84b0dbb7306f0db26a69dbcd89b56b90f9de502 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:976a8b2d06e3c6bfd12488297e8584dd704cc06c3e976c7e0f718c13e238e912, name=ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-osd-5-deactivate, maintainer=Guillaume Abrioux , ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.build-date=20240307, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, RELEASE=wip-yuri5-testing-2024-03-18-1144-e7e80a6) 2024-03-20T08:49:04.343 INFO:journalctl@ceph.osd.5.smithi194.stdout:Mar 20 08:49:04 smithi194 podman[63767]: 2024-03-20 08:49:04.200964146 +0000 UTC m=+0.653216354 container died 57d4132d927700dd02ae09c3d84b0dbb7306f0db26a69dbcd89b56b90f9de502 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:976a8b2d06e3c6bfd12488297e8584dd704cc06c3e976c7e0f718c13e238e912, name=ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-osd-5-deactivate, org.label-schema.build-date=20240307, maintainer=Guillaume Abrioux , RELEASE=wip-yuri5-testing-2024-03-18-1144-e7e80a6, org.label-schema.license=GPLv2, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-20T08:49:04.343 INFO:journalctl@ceph.osd.6.smithi194.stdout:Mar 20 08:49:04 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-osd-6[41285]: 2024-03-20T08:49:04.341+0000 7fabb4d64700 -1 osd.6 71 heartbeat_check: no reply from 172.21.15.183:6806 osd.0 since back 2024-03-20T08:48:18.168836+0000 front 2024-03-20T08:48:18.168757+0000 (oldest deadline 2024-03-20T08:48:40.468676+0000) 2024-03-20T08:49:04.343 INFO:journalctl@ceph.osd.6.smithi194.stdout:Mar 20 08:49:04 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-osd-6[41285]: 2024-03-20T08:49:04.341+0000 7fabb4d64700 -1 osd.6 71 heartbeat_check: no reply from 172.21.15.183:6814 osd.1 since back 2024-03-20T08:48:26.269930+0000 front 2024-03-20T08:48:26.269946+0000 (oldest deadline 2024-03-20T08:48:49.169783+0000) 2024-03-20T08:49:04.343 INFO:journalctl@ceph.osd.6.smithi194.stdout:Mar 20 08:49:04 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-osd-6[41285]: 2024-03-20T08:49:04.341+0000 7fabb4d64700 -1 osd.6 71 heartbeat_check: no reply from 172.21.15.183:6822 osd.2 since back 2024-03-20T08:48:35.571240+0000 front 2024-03-20T08:48:35.571309+0000 (oldest deadline 2024-03-20T08:49:01.471211+0000) 2024-03-20T08:49:04.895 DEBUG:teuthology.orchestra.run.smithi194:> sudo pkill -f 'journalctl -f -n 0 -u ceph-4c3f3eba-e695-11ee-95c9-87774f69a715@osd.5.service' 2024-03-20T08:49:04.937 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-20T08:49:04.938 INFO:tasks.cephadm.osd.5:Stopped osd.5 2024-03-20T08:49:04.938 INFO:tasks.cephadm.osd.6:Stopping osd.6... 2024-03-20T08:49:04.938 DEBUG:teuthology.orchestra.run.smithi194:> sudo systemctl stop ceph-4c3f3eba-e695-11ee-95c9-87774f69a715@osd.6 2024-03-20T08:49:05.138 INFO:journalctl@ceph.osd.6.smithi194.stdout:Mar 20 08:49:05 smithi194 systemd[1]: Stopping Ceph osd.6 for 4c3f3eba-e695-11ee-95c9-87774f69a715... 2024-03-20T08:49:05.139 INFO:journalctl@ceph.osd.7.smithi194.stdout:Mar 20 08:49:04 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-osd-7[44872]: 2024-03-20T08:49:04.830+0000 7f37d5961700 -1 osd.7 71 heartbeat_check: no reply from 172.21.15.183:6806 osd.0 since back 2024-03-20T08:48:16.746885+0000 front 2024-03-20T08:48:16.746994+0000 (oldest deadline 2024-03-20T08:48:42.046867+0000) 2024-03-20T08:49:05.139 INFO:journalctl@ceph.osd.7.smithi194.stdout:Mar 20 08:49:04 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-osd-7[44872]: 2024-03-20T08:49:04.830+0000 7f37d5961700 -1 osd.7 71 heartbeat_check: no reply from 172.21.15.183:6814 osd.1 since back 2024-03-20T08:48:26.147608+0000 front 2024-03-20T08:48:26.147671+0000 (oldest deadline 2024-03-20T08:48:50.847640+0000) 2024-03-20T08:49:05.139 INFO:journalctl@ceph.osd.7.smithi194.stdout:Mar 20 08:49:04 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-osd-7[44872]: 2024-03-20T08:49:04.830+0000 7f37d5961700 -1 osd.7 71 heartbeat_check: no reply from 172.21.15.183:6822 osd.2 since back 2024-03-20T08:48:36.148515+0000 front 2024-03-20T08:48:36.148487+0000 (oldest deadline 2024-03-20T08:48:59.048455+0000) 2024-03-20T08:49:05.139 INFO:journalctl@ceph.osd.7.smithi194.stdout:Mar 20 08:49:04 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-osd-7[44872]: 2024-03-20T08:49:04.831+0000 7f37d5961700 -1 osd.7 71 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.14556.0:149 2.0 2:353ec8f9:::gateway.conf:head [getxattr epoch in=5b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e71) 2024-03-20T08:49:05.639 INFO:journalctl@ceph.osd.6.smithi194.stdout:Mar 20 08:49:05 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-osd-6[41285]: 2024-03-20T08:49:05.247+0000 7fabbc778700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.6 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-20T08:49:05.639 INFO:journalctl@ceph.osd.6.smithi194.stdout:Mar 20 08:49:05 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-osd-6[41285]: 2024-03-20T08:49:05.247+0000 7fabbc778700 -1 osd.6 71 *** Got signal Terminated *** 2024-03-20T08:49:05.639 INFO:journalctl@ceph.osd.6.smithi194.stdout:Mar 20 08:49:05 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-osd-6[41285]: 2024-03-20T08:49:05.247+0000 7fabbc778700 -1 osd.6 71 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-20T08:49:05.639 INFO:journalctl@ceph.osd.6.smithi194.stdout:Mar 20 08:49:05 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-osd-6[41285]: 2024-03-20T08:49:05.366+0000 7fabb4d64700 -1 osd.6 71 heartbeat_check: no reply from 172.21.15.183:6806 osd.0 since back 2024-03-20T08:48:18.168836+0000 front 2024-03-20T08:48:18.168757+0000 (oldest deadline 2024-03-20T08:48:40.468676+0000) 2024-03-20T08:49:05.639 INFO:journalctl@ceph.osd.6.smithi194.stdout:Mar 20 08:49:05 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-osd-6[41285]: 2024-03-20T08:49:05.366+0000 7fabb4d64700 -1 osd.6 71 heartbeat_check: no reply from 172.21.15.183:6814 osd.1 since back 2024-03-20T08:48:26.269930+0000 front 2024-03-20T08:48:26.269946+0000 (oldest deadline 2024-03-20T08:48:49.169783+0000) 2024-03-20T08:49:05.639 INFO:journalctl@ceph.osd.6.smithi194.stdout:Mar 20 08:49:05 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-osd-6[41285]: 2024-03-20T08:49:05.366+0000 7fabb4d64700 -1 osd.6 71 heartbeat_check: no reply from 172.21.15.183:6822 osd.2 since back 2024-03-20T08:48:35.571240+0000 front 2024-03-20T08:48:35.571309+0000 (oldest deadline 2024-03-20T08:49:01.471211+0000) 2024-03-20T08:49:06.139 INFO:journalctl@ceph.osd.7.smithi194.stdout:Mar 20 08:49:05 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-osd-7[44872]: 2024-03-20T08:49:05.866+0000 7f37d5961700 -1 osd.7 71 heartbeat_check: no reply from 172.21.15.183:6806 osd.0 since back 2024-03-20T08:48:16.746885+0000 front 2024-03-20T08:48:16.746994+0000 (oldest deadline 2024-03-20T08:48:42.046867+0000) 2024-03-20T08:49:06.139 INFO:journalctl@ceph.osd.7.smithi194.stdout:Mar 20 08:49:05 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-osd-7[44872]: 2024-03-20T08:49:05.866+0000 7f37d5961700 -1 osd.7 71 heartbeat_check: no reply from 172.21.15.183:6814 osd.1 since back 2024-03-20T08:48:26.147608+0000 front 2024-03-20T08:48:26.147671+0000 (oldest deadline 2024-03-20T08:48:50.847640+0000) 2024-03-20T08:49:06.139 INFO:journalctl@ceph.osd.7.smithi194.stdout:Mar 20 08:49:05 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-osd-7[44872]: 2024-03-20T08:49:05.866+0000 7f37d5961700 -1 osd.7 71 heartbeat_check: no reply from 172.21.15.183:6822 osd.2 since back 2024-03-20T08:48:36.148515+0000 front 2024-03-20T08:48:36.148487+0000 (oldest deadline 2024-03-20T08:48:59.048455+0000) 2024-03-20T08:49:06.139 INFO:journalctl@ceph.osd.7.smithi194.stdout:Mar 20 08:49:05 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-osd-7[44872]: 2024-03-20T08:49:05.866+0000 7f37d5961700 -1 osd.7 71 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.14556.0:149 2.0 2:353ec8f9:::gateway.conf:head [getxattr epoch in=5b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e71) 2024-03-20T08:49:06.639 INFO:journalctl@ceph.osd.6.smithi194.stdout:Mar 20 08:49:06 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-osd-6[41285]: 2024-03-20T08:49:06.355+0000 7fabb4d64700 -1 osd.6 71 heartbeat_check: no reply from 172.21.15.183:6806 osd.0 since back 2024-03-20T08:48:18.168836+0000 front 2024-03-20T08:48:18.168757+0000 (oldest deadline 2024-03-20T08:48:40.468676+0000) 2024-03-20T08:49:06.639 INFO:journalctl@ceph.osd.6.smithi194.stdout:Mar 20 08:49:06 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-osd-6[41285]: 2024-03-20T08:49:06.355+0000 7fabb4d64700 -1 osd.6 71 heartbeat_check: no reply from 172.21.15.183:6814 osd.1 since back 2024-03-20T08:48:26.269930+0000 front 2024-03-20T08:48:26.269946+0000 (oldest deadline 2024-03-20T08:48:49.169783+0000) 2024-03-20T08:49:06.639 INFO:journalctl@ceph.osd.6.smithi194.stdout:Mar 20 08:49:06 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-osd-6[41285]: 2024-03-20T08:49:06.355+0000 7fabb4d64700 -1 osd.6 71 heartbeat_check: no reply from 172.21.15.183:6822 osd.2 since back 2024-03-20T08:48:35.571240+0000 front 2024-03-20T08:48:35.571309+0000 (oldest deadline 2024-03-20T08:49:01.471211+0000) 2024-03-20T08:49:07.139 INFO:journalctl@ceph.osd.7.smithi194.stdout:Mar 20 08:49:06 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-osd-7[44872]: 2024-03-20T08:49:06.828+0000 7f37d5961700 -1 osd.7 71 heartbeat_check: no reply from 172.21.15.183:6806 osd.0 since back 2024-03-20T08:48:16.746885+0000 front 2024-03-20T08:48:16.746994+0000 (oldest deadline 2024-03-20T08:48:42.046867+0000) 2024-03-20T08:49:07.139 INFO:journalctl@ceph.osd.7.smithi194.stdout:Mar 20 08:49:06 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-osd-7[44872]: 2024-03-20T08:49:06.828+0000 7f37d5961700 -1 osd.7 71 heartbeat_check: no reply from 172.21.15.183:6814 osd.1 since back 2024-03-20T08:48:26.147608+0000 front 2024-03-20T08:48:26.147671+0000 (oldest deadline 2024-03-20T08:48:50.847640+0000) 2024-03-20T08:49:07.139 INFO:journalctl@ceph.osd.7.smithi194.stdout:Mar 20 08:49:06 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-osd-7[44872]: 2024-03-20T08:49:06.828+0000 7f37d5961700 -1 osd.7 71 heartbeat_check: no reply from 172.21.15.183:6822 osd.2 since back 2024-03-20T08:48:36.148515+0000 front 2024-03-20T08:48:36.148487+0000 (oldest deadline 2024-03-20T08:48:59.048455+0000) 2024-03-20T08:49:07.139 INFO:journalctl@ceph.osd.7.smithi194.stdout:Mar 20 08:49:06 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-osd-7[44872]: 2024-03-20T08:49:06.828+0000 7f37d5961700 -1 osd.7 71 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.14556.0:149 2.0 2:353ec8f9:::gateway.conf:head [getxattr epoch in=5b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e71) 2024-03-20T08:49:07.577 INFO:journalctl@ceph.osd.6.smithi194.stdout:Mar 20 08:49:07 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-osd-6[41285]: 2024-03-20T08:49:07.313+0000 7fabb4d64700 -1 osd.6 71 heartbeat_check: no reply from 172.21.15.183:6806 osd.0 since back 2024-03-20T08:48:18.168836+0000 front 2024-03-20T08:48:18.168757+0000 (oldest deadline 2024-03-20T08:48:40.468676+0000) 2024-03-20T08:49:07.577 INFO:journalctl@ceph.osd.6.smithi194.stdout:Mar 20 08:49:07 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-osd-6[41285]: 2024-03-20T08:49:07.313+0000 7fabb4d64700 -1 osd.6 71 heartbeat_check: no reply from 172.21.15.183:6814 osd.1 since back 2024-03-20T08:48:26.269930+0000 front 2024-03-20T08:48:26.269946+0000 (oldest deadline 2024-03-20T08:48:49.169783+0000) 2024-03-20T08:49:07.577 INFO:journalctl@ceph.osd.6.smithi194.stdout:Mar 20 08:49:07 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-osd-6[41285]: 2024-03-20T08:49:07.313+0000 7fabb4d64700 -1 osd.6 71 heartbeat_check: no reply from 172.21.15.183:6822 osd.2 since back 2024-03-20T08:48:35.571240+0000 front 2024-03-20T08:48:35.571309+0000 (oldest deadline 2024-03-20T08:49:01.471211+0000) 2024-03-20T08:49:08.139 INFO:journalctl@ceph.osd.7.smithi194.stdout:Mar 20 08:49:07 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-osd-7[44872]: 2024-03-20T08:49:07.871+0000 7f37d5961700 -1 osd.7 71 heartbeat_check: no reply from 172.21.15.183:6806 osd.0 since back 2024-03-20T08:48:16.746885+0000 front 2024-03-20T08:48:16.746994+0000 (oldest deadline 2024-03-20T08:48:42.046867+0000) 2024-03-20T08:49:08.139 INFO:journalctl@ceph.osd.7.smithi194.stdout:Mar 20 08:49:07 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-osd-7[44872]: 2024-03-20T08:49:07.871+0000 7f37d5961700 -1 osd.7 71 heartbeat_check: no reply from 172.21.15.183:6814 osd.1 since back 2024-03-20T08:48:26.147608+0000 front 2024-03-20T08:48:26.147671+0000 (oldest deadline 2024-03-20T08:48:50.847640+0000) 2024-03-20T08:49:08.139 INFO:journalctl@ceph.osd.7.smithi194.stdout:Mar 20 08:49:07 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-osd-7[44872]: 2024-03-20T08:49:07.871+0000 7f37d5961700 -1 osd.7 71 heartbeat_check: no reply from 172.21.15.183:6822 osd.2 since back 2024-03-20T08:48:36.148515+0000 front 2024-03-20T08:48:36.148487+0000 (oldest deadline 2024-03-20T08:48:59.048455+0000) 2024-03-20T08:49:08.139 INFO:journalctl@ceph.osd.7.smithi194.stdout:Mar 20 08:49:07 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-osd-7[44872]: 2024-03-20T08:49:07.871+0000 7f37d5961700 -1 osd.7 71 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.14556.0:149 2.0 2:353ec8f9:::gateway.conf:head [getxattr epoch in=5b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e71) 2024-03-20T08:49:08.639 INFO:journalctl@ceph.osd.6.smithi194.stdout:Mar 20 08:49:08 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-osd-6[41285]: 2024-03-20T08:49:08.273+0000 7fabb4d64700 -1 osd.6 71 heartbeat_check: no reply from 172.21.15.183:6806 osd.0 since back 2024-03-20T08:48:18.168836+0000 front 2024-03-20T08:48:18.168757+0000 (oldest deadline 2024-03-20T08:48:40.468676+0000) 2024-03-20T08:49:08.639 INFO:journalctl@ceph.osd.6.smithi194.stdout:Mar 20 08:49:08 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-osd-6[41285]: 2024-03-20T08:49:08.273+0000 7fabb4d64700 -1 osd.6 71 heartbeat_check: no reply from 172.21.15.183:6814 osd.1 since back 2024-03-20T08:48:26.269930+0000 front 2024-03-20T08:48:26.269946+0000 (oldest deadline 2024-03-20T08:48:49.169783+0000) 2024-03-20T08:49:08.639 INFO:journalctl@ceph.osd.6.smithi194.stdout:Mar 20 08:49:08 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-osd-6[41285]: 2024-03-20T08:49:08.273+0000 7fabb4d64700 -1 osd.6 71 heartbeat_check: no reply from 172.21.15.183:6822 osd.2 since back 2024-03-20T08:48:35.571240+0000 front 2024-03-20T08:48:35.571309+0000 (oldest deadline 2024-03-20T08:49:01.471211+0000) 2024-03-20T08:49:09.139 INFO:journalctl@ceph.osd.7.smithi194.stdout:Mar 20 08:49:08 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-osd-7[44872]: 2024-03-20T08:49:08.871+0000 7f37d5961700 -1 osd.7 71 heartbeat_check: no reply from 172.21.15.183:6806 osd.0 since back 2024-03-20T08:48:16.746885+0000 front 2024-03-20T08:48:16.746994+0000 (oldest deadline 2024-03-20T08:48:42.046867+0000) 2024-03-20T08:49:09.139 INFO:journalctl@ceph.osd.7.smithi194.stdout:Mar 20 08:49:08 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-osd-7[44872]: 2024-03-20T08:49:08.871+0000 7f37d5961700 -1 osd.7 71 heartbeat_check: no reply from 172.21.15.183:6814 osd.1 since back 2024-03-20T08:48:26.147608+0000 front 2024-03-20T08:48:26.147671+0000 (oldest deadline 2024-03-20T08:48:50.847640+0000) 2024-03-20T08:49:09.139 INFO:journalctl@ceph.osd.7.smithi194.stdout:Mar 20 08:49:08 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-osd-7[44872]: 2024-03-20T08:49:08.871+0000 7f37d5961700 -1 osd.7 71 heartbeat_check: no reply from 172.21.15.183:6822 osd.2 since back 2024-03-20T08:48:36.148515+0000 front 2024-03-20T08:48:36.148487+0000 (oldest deadline 2024-03-20T08:48:59.048455+0000) 2024-03-20T08:49:09.139 INFO:journalctl@ceph.osd.7.smithi194.stdout:Mar 20 08:49:08 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-osd-7[44872]: 2024-03-20T08:49:08.871+0000 7f37d5961700 -1 osd.7 71 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.14556.0:149 2.0 2:353ec8f9:::gateway.conf:head [getxattr epoch in=5b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e71) 2024-03-20T08:49:09.638 INFO:journalctl@ceph.osd.6.smithi194.stdout:Mar 20 08:49:09 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-osd-6[41285]: 2024-03-20T08:49:09.254+0000 7fabb4d64700 -1 osd.6 71 heartbeat_check: no reply from 172.21.15.183:6806 osd.0 since back 2024-03-20T08:48:18.168836+0000 front 2024-03-20T08:48:18.168757+0000 (oldest deadline 2024-03-20T08:48:40.468676+0000) 2024-03-20T08:49:09.639 INFO:journalctl@ceph.osd.6.smithi194.stdout:Mar 20 08:49:09 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-osd-6[41285]: 2024-03-20T08:49:09.254+0000 7fabb4d64700 -1 osd.6 71 heartbeat_check: no reply from 172.21.15.183:6814 osd.1 since back 2024-03-20T08:48:26.269930+0000 front 2024-03-20T08:48:26.269946+0000 (oldest deadline 2024-03-20T08:48:49.169783+0000) 2024-03-20T08:49:09.639 INFO:journalctl@ceph.osd.6.smithi194.stdout:Mar 20 08:49:09 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-osd-6[41285]: 2024-03-20T08:49:09.254+0000 7fabb4d64700 -1 osd.6 71 heartbeat_check: no reply from 172.21.15.183:6822 osd.2 since back 2024-03-20T08:48:35.571240+0000 front 2024-03-20T08:48:35.571309+0000 (oldest deadline 2024-03-20T08:49:01.471211+0000) 2024-03-20T08:49:10.389 INFO:journalctl@ceph.osd.7.smithi194.stdout:Mar 20 08:49:09 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-osd-7[44872]: 2024-03-20T08:49:09.910+0000 7f37d5961700 -1 osd.7 71 heartbeat_check: no reply from 172.21.15.183:6806 osd.0 since back 2024-03-20T08:48:16.746885+0000 front 2024-03-20T08:48:16.746994+0000 (oldest deadline 2024-03-20T08:48:42.046867+0000) 2024-03-20T08:49:10.389 INFO:journalctl@ceph.osd.7.smithi194.stdout:Mar 20 08:49:09 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-osd-7[44872]: 2024-03-20T08:49:09.910+0000 7f37d5961700 -1 osd.7 71 heartbeat_check: no reply from 172.21.15.183:6814 osd.1 since back 2024-03-20T08:48:26.147608+0000 front 2024-03-20T08:48:26.147671+0000 (oldest deadline 2024-03-20T08:48:50.847640+0000) 2024-03-20T08:49:10.389 INFO:journalctl@ceph.osd.7.smithi194.stdout:Mar 20 08:49:09 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-osd-7[44872]: 2024-03-20T08:49:09.910+0000 7f37d5961700 -1 osd.7 71 heartbeat_check: no reply from 172.21.15.183:6822 osd.2 since back 2024-03-20T08:48:36.148515+0000 front 2024-03-20T08:48:36.148487+0000 (oldest deadline 2024-03-20T08:48:59.048455+0000) 2024-03-20T08:49:10.389 INFO:journalctl@ceph.osd.7.smithi194.stdout:Mar 20 08:49:09 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-osd-7[44872]: 2024-03-20T08:49:09.910+0000 7f37d5961700 -1 osd.7 71 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.14556.0:149 2.0 2:353ec8f9:::gateway.conf:head [getxattr epoch in=5b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e71) 2024-03-20T08:49:10.907 INFO:journalctl@ceph.osd.6.smithi194.stdout:Mar 20 08:49:10 smithi194 podman[63873]: 2024-03-20 08:49:10.645064697 +0000 UTC m=+5.494469348 container died 9a38fbc0b31e6ff724a7f28167a48478c9bac774f3bcb5a9d52e5d8c25db9934 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:976a8b2d06e3c6bfd12488297e8584dd704cc06c3e976c7e0f718c13e238e912, name=ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-osd-6, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, ceph=True, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, RELEASE=wip-yuri5-testing-2024-03-18-1144-e7e80a6, org.label-schema.build-date=20240307, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-20T08:49:11.266 INFO:journalctl@ceph.osd.6.smithi194.stdout:Mar 20 08:49:11 smithi194 podman[63873]: 2024-03-20 08:49:11.073196463 +0000 UTC m=+5.922601102 container cleanup 9a38fbc0b31e6ff724a7f28167a48478c9bac774f3bcb5a9d52e5d8c25db9934 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:976a8b2d06e3c6bfd12488297e8584dd704cc06c3e976c7e0f718c13e238e912, name=ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-osd-6, GIT_CLEAN=True, org.label-schema.build-date=20240307, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, ceph=True, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-yuri5-testing-2024-03-18-1144-e7e80a6) 2024-03-20T08:49:11.266 INFO:journalctl@ceph.osd.6.smithi194.stdout:Mar 20 08:49:11 smithi194 bash[63873]: ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-osd-6 2024-03-20T08:49:11.266 INFO:journalctl@ceph.osd.7.smithi194.stdout:Mar 20 08:49:10 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-osd-7[44872]: 2024-03-20T08:49:10.918+0000 7f37d5961700 -1 osd.7 71 heartbeat_check: no reply from 172.21.15.183:6806 osd.0 since back 2024-03-20T08:48:16.746885+0000 front 2024-03-20T08:48:16.746994+0000 (oldest deadline 2024-03-20T08:48:42.046867+0000) 2024-03-20T08:49:11.267 INFO:journalctl@ceph.osd.7.smithi194.stdout:Mar 20 08:49:10 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-osd-7[44872]: 2024-03-20T08:49:10.918+0000 7f37d5961700 -1 osd.7 71 heartbeat_check: no reply from 172.21.15.183:6814 osd.1 since back 2024-03-20T08:48:26.147608+0000 front 2024-03-20T08:48:26.147671+0000 (oldest deadline 2024-03-20T08:48:50.847640+0000) 2024-03-20T08:49:11.267 INFO:journalctl@ceph.osd.7.smithi194.stdout:Mar 20 08:49:10 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-osd-7[44872]: 2024-03-20T08:49:10.918+0000 7f37d5961700 -1 osd.7 71 heartbeat_check: no reply from 172.21.15.183:6822 osd.2 since back 2024-03-20T08:48:36.148515+0000 front 2024-03-20T08:48:36.148487+0000 (oldest deadline 2024-03-20T08:48:59.048455+0000) 2024-03-20T08:49:11.267 INFO:journalctl@ceph.osd.7.smithi194.stdout:Mar 20 08:49:10 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-osd-7[44872]: 2024-03-20T08:49:10.918+0000 7f37d5961700 -1 osd.7 71 heartbeat_check: no reply from 172.21.15.183:6830 osd.3 since back 2024-03-20T08:48:44.949598+0000 front 2024-03-20T08:48:44.949663+0000 (oldest deadline 2024-03-20T08:49:10.849451+0000) 2024-03-20T08:49:11.267 INFO:journalctl@ceph.osd.7.smithi194.stdout:Mar 20 08:49:10 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-osd-7[44872]: 2024-03-20T08:49:10.918+0000 7f37d5961700 -1 osd.7 71 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.14556.0:149 2.0 2:353ec8f9:::gateway.conf:head [getxattr epoch in=5b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e71) 2024-03-20T08:49:11.638 INFO:journalctl@ceph.osd.6.smithi194.stdout:Mar 20 08:49:11 smithi194 podman[63890]: 2024-03-20 08:49:11.265973746 +0000 UTC m=+0.612029677 container remove 9a38fbc0b31e6ff724a7f28167a48478c9bac774f3bcb5a9d52e5d8c25db9934 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:976a8b2d06e3c6bfd12488297e8584dd704cc06c3e976c7e0f718c13e238e912, name=ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-osd-6, GIT_CLEAN=True, RELEASE=wip-yuri5-testing-2024-03-18-1144-e7e80a6, org.label-schema.build-date=20240307, maintainer=Guillaume Abrioux , ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-20T08:49:11.977 INFO:journalctl@ceph.osd.6.smithi194.stdout:Mar 20 08:49:11 smithi194 podman[63979]: 2024-03-20 08:49:11.588557421 +0000 UTC m=+0.017538231 image pull 9ceba81fb2c8d8682cb159527017d591637b153b3aa4a390ffa8586db6dcda00 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:976a8b2d06e3c6bfd12488297e8584dd704cc06c3e976c7e0f718c13e238e912 2024-03-20T08:49:11.977 INFO:journalctl@ceph.osd.6.smithi194.stdout:Mar 20 08:49:11 smithi194 podman[63979]: 2024-03-20 08:49:11.708633485 +0000 UTC m=+0.137614319 container create 065e782cddc2c0f94a0e81ea1dd1e69fdbc95116c2ac05845388199673dc0fad (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:976a8b2d06e3c6bfd12488297e8584dd704cc06c3e976c7e0f718c13e238e912, name=ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-osd-6-deactivate, org.label-schema.build-date=20240307, ceph=True, org.label-schema.license=GPLv2, GIT_CLEAN=True, RELEASE=wip-yuri5-testing-2024-03-18-1144-e7e80a6, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0) 2024-03-20T08:49:11.978 INFO:journalctl@ceph.osd.6.smithi194.stdout:Mar 20 08:49:11 smithi194 podman[63979]: 2024-03-20 08:49:11.902532253 +0000 UTC m=+0.331513047 container init 065e782cddc2c0f94a0e81ea1dd1e69fdbc95116c2ac05845388199673dc0fad (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:976a8b2d06e3c6bfd12488297e8584dd704cc06c3e976c7e0f718c13e238e912, name=ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-osd-6-deactivate, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , RELEASE=wip-yuri5-testing-2024-03-18-1144-e7e80a6, GIT_BRANCH=HEAD, org.label-schema.build-date=20240307, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_CLEAN=True) 2024-03-20T08:49:11.978 INFO:journalctl@ceph.osd.6.smithi194.stdout:Mar 20 08:49:11 smithi194 podman[63979]: 2024-03-20 08:49:11.905657107 +0000 UTC m=+0.334637904 container start 065e782cddc2c0f94a0e81ea1dd1e69fdbc95116c2ac05845388199673dc0fad (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:976a8b2d06e3c6bfd12488297e8584dd704cc06c3e976c7e0f718c13e238e912, name=ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-osd-6-deactivate, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, RELEASE=wip-yuri5-testing-2024-03-18-1144-e7e80a6, org.label-schema.build-date=20240307, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2) 2024-03-20T08:49:11.978 INFO:journalctl@ceph.osd.6.smithi194.stdout:Mar 20 08:49:11 smithi194 podman[63979]: 2024-03-20 08:49:11.977508047 +0000 UTC m=+0.406488849 container attach 065e782cddc2c0f94a0e81ea1dd1e69fdbc95116c2ac05845388199673dc0fad (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:976a8b2d06e3c6bfd12488297e8584dd704cc06c3e976c7e0f718c13e238e912, name=ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-osd-6-deactivate, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240307, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, ceph=True, GIT_CLEAN=True, RELEASE=wip-yuri5-testing-2024-03-18-1144-e7e80a6, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-20T08:49:11.978 INFO:journalctl@ceph.osd.7.smithi194.stdout:Mar 20 08:49:11 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-osd-7[44872]: 2024-03-20T08:49:11.869+0000 7f37d5961700 -1 osd.7 71 heartbeat_check: no reply from 172.21.15.183:6806 osd.0 since back 2024-03-20T08:48:16.746885+0000 front 2024-03-20T08:48:16.746994+0000 (oldest deadline 2024-03-20T08:48:42.046867+0000) 2024-03-20T08:49:11.978 INFO:journalctl@ceph.osd.7.smithi194.stdout:Mar 20 08:49:11 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-osd-7[44872]: 2024-03-20T08:49:11.869+0000 7f37d5961700 -1 osd.7 71 heartbeat_check: no reply from 172.21.15.183:6814 osd.1 since back 2024-03-20T08:48:26.147608+0000 front 2024-03-20T08:48:26.147671+0000 (oldest deadline 2024-03-20T08:48:50.847640+0000) 2024-03-20T08:49:11.978 INFO:journalctl@ceph.osd.7.smithi194.stdout:Mar 20 08:49:11 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-osd-7[44872]: 2024-03-20T08:49:11.869+0000 7f37d5961700 -1 osd.7 71 heartbeat_check: no reply from 172.21.15.183:6822 osd.2 since back 2024-03-20T08:48:36.148515+0000 front 2024-03-20T08:48:36.148487+0000 (oldest deadline 2024-03-20T08:48:59.048455+0000) 2024-03-20T08:49:11.978 INFO:journalctl@ceph.osd.7.smithi194.stdout:Mar 20 08:49:11 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-osd-7[44872]: 2024-03-20T08:49:11.869+0000 7f37d5961700 -1 osd.7 71 heartbeat_check: no reply from 172.21.15.183:6830 osd.3 since back 2024-03-20T08:48:44.949598+0000 front 2024-03-20T08:48:44.949663+0000 (oldest deadline 2024-03-20T08:49:10.849451+0000) 2024-03-20T08:49:11.978 INFO:journalctl@ceph.osd.7.smithi194.stdout:Mar 20 08:49:11 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-osd-7[44872]: 2024-03-20T08:49:11.869+0000 7f37d5961700 -1 osd.7 71 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.14556.0:149 2.0 2:353ec8f9:::gateway.conf:head [getxattr epoch in=5b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e71) 2024-03-20T08:49:12.230 INFO:journalctl@ceph.osd.6.smithi194.stdout:Mar 20 08:49:12 smithi194 podman[63979]: 2024-03-20 08:49:12.202014863 +0000 UTC m=+0.630995653 container died 065e782cddc2c0f94a0e81ea1dd1e69fdbc95116c2ac05845388199673dc0fad (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:976a8b2d06e3c6bfd12488297e8584dd704cc06c3e976c7e0f718c13e238e912, name=ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-osd-6-deactivate, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-yuri5-testing-2024-03-18-1144-e7e80a6, io.buildah.version=1.33.5, ceph=True, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240307) 2024-03-20T08:49:12.929 DEBUG:teuthology.orchestra.run.smithi194:> sudo pkill -f 'journalctl -f -n 0 -u ceph-4c3f3eba-e695-11ee-95c9-87774f69a715@osd.6.service' 2024-03-20T08:49:12.971 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-20T08:49:12.971 INFO:tasks.cephadm.osd.6:Stopped osd.6 2024-03-20T08:49:12.971 INFO:tasks.cephadm.osd.7:Stopping osd.7... 2024-03-20T08:49:12.971 DEBUG:teuthology.orchestra.run.smithi194:> sudo systemctl stop ceph-4c3f3eba-e695-11ee-95c9-87774f69a715@osd.7 2024-03-20T08:49:13.138 INFO:journalctl@ceph.osd.7.smithi194.stdout:Mar 20 08:49:12 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-osd-7[44872]: 2024-03-20T08:49:12.850+0000 7f37d5961700 -1 osd.7 71 heartbeat_check: no reply from 172.21.15.183:6806 osd.0 since back 2024-03-20T08:48:16.746885+0000 front 2024-03-20T08:48:16.746994+0000 (oldest deadline 2024-03-20T08:48:42.046867+0000) 2024-03-20T08:49:13.139 INFO:journalctl@ceph.osd.7.smithi194.stdout:Mar 20 08:49:12 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-osd-7[44872]: 2024-03-20T08:49:12.850+0000 7f37d5961700 -1 osd.7 71 heartbeat_check: no reply from 172.21.15.183:6814 osd.1 since back 2024-03-20T08:48:26.147608+0000 front 2024-03-20T08:48:26.147671+0000 (oldest deadline 2024-03-20T08:48:50.847640+0000) 2024-03-20T08:49:13.139 INFO:journalctl@ceph.osd.7.smithi194.stdout:Mar 20 08:49:12 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-osd-7[44872]: 2024-03-20T08:49:12.850+0000 7f37d5961700 -1 osd.7 71 heartbeat_check: no reply from 172.21.15.183:6822 osd.2 since back 2024-03-20T08:48:36.148515+0000 front 2024-03-20T08:48:36.148487+0000 (oldest deadline 2024-03-20T08:48:59.048455+0000) 2024-03-20T08:49:13.139 INFO:journalctl@ceph.osd.7.smithi194.stdout:Mar 20 08:49:12 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-osd-7[44872]: 2024-03-20T08:49:12.851+0000 7f37d5961700 -1 osd.7 71 heartbeat_check: no reply from 172.21.15.183:6830 osd.3 since back 2024-03-20T08:48:44.949598+0000 front 2024-03-20T08:48:44.949663+0000 (oldest deadline 2024-03-20T08:49:10.849451+0000) 2024-03-20T08:49:13.139 INFO:journalctl@ceph.osd.7.smithi194.stdout:Mar 20 08:49:12 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-osd-7[44872]: 2024-03-20T08:49:12.851+0000 7f37d5961700 -1 osd.7 71 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.14556.0:149 2.0 2:353ec8f9:::gateway.conf:head [getxattr epoch in=5b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e71) 2024-03-20T08:49:13.139 INFO:journalctl@ceph.osd.7.smithi194.stdout:Mar 20 08:49:13 smithi194 systemd[1]: Stopping Ceph osd.7 for 4c3f3eba-e695-11ee-95c9-87774f69a715... 2024-03-20T08:49:13.639 INFO:journalctl@ceph.osd.7.smithi194.stdout:Mar 20 08:49:13 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-osd-7[44872]: 2024-03-20T08:49:13.317+0000 7f37dd375700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.7 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-20T08:49:13.639 INFO:journalctl@ceph.osd.7.smithi194.stdout:Mar 20 08:49:13 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-osd-7[44872]: 2024-03-20T08:49:13.317+0000 7f37dd375700 -1 osd.7 71 *** Got signal Terminated *** 2024-03-20T08:49:13.639 INFO:journalctl@ceph.osd.7.smithi194.stdout:Mar 20 08:49:13 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-osd-7[44872]: 2024-03-20T08:49:13.317+0000 7f37dd375700 -1 osd.7 71 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-20T08:49:14.139 INFO:journalctl@ceph.osd.7.smithi194.stdout:Mar 20 08:49:13 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-osd-7[44872]: 2024-03-20T08:49:13.821+0000 7f37d5961700 -1 osd.7 71 heartbeat_check: no reply from 172.21.15.183:6806 osd.0 since back 2024-03-20T08:48:16.746885+0000 front 2024-03-20T08:48:16.746994+0000 (oldest deadline 2024-03-20T08:48:42.046867+0000) 2024-03-20T08:49:14.139 INFO:journalctl@ceph.osd.7.smithi194.stdout:Mar 20 08:49:13 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-osd-7[44872]: 2024-03-20T08:49:13.821+0000 7f37d5961700 -1 osd.7 71 heartbeat_check: no reply from 172.21.15.183:6814 osd.1 since back 2024-03-20T08:48:26.147608+0000 front 2024-03-20T08:48:26.147671+0000 (oldest deadline 2024-03-20T08:48:50.847640+0000) 2024-03-20T08:49:14.139 INFO:journalctl@ceph.osd.7.smithi194.stdout:Mar 20 08:49:13 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-osd-7[44872]: 2024-03-20T08:49:13.821+0000 7f37d5961700 -1 osd.7 71 heartbeat_check: no reply from 172.21.15.183:6822 osd.2 since back 2024-03-20T08:48:36.148515+0000 front 2024-03-20T08:48:36.148487+0000 (oldest deadline 2024-03-20T08:48:59.048455+0000) 2024-03-20T08:49:14.139 INFO:journalctl@ceph.osd.7.smithi194.stdout:Mar 20 08:49:13 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-osd-7[44872]: 2024-03-20T08:49:13.821+0000 7f37d5961700 -1 osd.7 71 heartbeat_check: no reply from 172.21.15.183:6830 osd.3 since back 2024-03-20T08:48:44.949598+0000 front 2024-03-20T08:48:44.949663+0000 (oldest deadline 2024-03-20T08:49:10.849451+0000) 2024-03-20T08:49:14.139 INFO:journalctl@ceph.osd.7.smithi194.stdout:Mar 20 08:49:13 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-osd-7[44872]: 2024-03-20T08:49:13.821+0000 7f37d5961700 -1 osd.7 71 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.14556.0:149 2.0 2:353ec8f9:::gateway.conf:head [getxattr epoch in=5b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e71) 2024-03-20T08:49:15.142 INFO:journalctl@ceph.osd.7.smithi194.stdout:Mar 20 08:49:14 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-osd-7[44872]: 2024-03-20T08:49:14.849+0000 7f37d5961700 -1 osd.7 71 heartbeat_check: no reply from 172.21.15.183:6806 osd.0 since back 2024-03-20T08:48:16.746885+0000 front 2024-03-20T08:48:16.746994+0000 (oldest deadline 2024-03-20T08:48:42.046867+0000) 2024-03-20T08:49:15.142 INFO:journalctl@ceph.osd.7.smithi194.stdout:Mar 20 08:49:14 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-osd-7[44872]: 2024-03-20T08:49:14.849+0000 7f37d5961700 -1 osd.7 71 heartbeat_check: no reply from 172.21.15.183:6814 osd.1 since back 2024-03-20T08:48:26.147608+0000 front 2024-03-20T08:48:26.147671+0000 (oldest deadline 2024-03-20T08:48:50.847640+0000) 2024-03-20T08:49:15.142 INFO:journalctl@ceph.osd.7.smithi194.stdout:Mar 20 08:49:14 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-osd-7[44872]: 2024-03-20T08:49:14.849+0000 7f37d5961700 -1 osd.7 71 heartbeat_check: no reply from 172.21.15.183:6822 osd.2 since back 2024-03-20T08:48:36.148515+0000 front 2024-03-20T08:48:36.148487+0000 (oldest deadline 2024-03-20T08:48:59.048455+0000) 2024-03-20T08:49:15.142 INFO:journalctl@ceph.osd.7.smithi194.stdout:Mar 20 08:49:14 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-osd-7[44872]: 2024-03-20T08:49:14.849+0000 7f37d5961700 -1 osd.7 71 heartbeat_check: no reply from 172.21.15.183:6830 osd.3 since back 2024-03-20T08:48:44.949598+0000 front 2024-03-20T08:48:44.949663+0000 (oldest deadline 2024-03-20T08:49:10.849451+0000) 2024-03-20T08:49:15.142 INFO:journalctl@ceph.osd.7.smithi194.stdout:Mar 20 08:49:14 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-osd-7[44872]: 2024-03-20T08:49:14.850+0000 7f37d5961700 -1 osd.7 71 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.14556.0:149 2.0 2:353ec8f9:::gateway.conf:head [getxattr epoch in=5b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e71) 2024-03-20T08:49:16.139 INFO:journalctl@ceph.osd.7.smithi194.stdout:Mar 20 08:49:15 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-osd-7[44872]: 2024-03-20T08:49:15.861+0000 7f37d5961700 -1 osd.7 71 heartbeat_check: no reply from 172.21.15.183:6806 osd.0 since back 2024-03-20T08:48:16.746885+0000 front 2024-03-20T08:48:16.746994+0000 (oldest deadline 2024-03-20T08:48:42.046867+0000) 2024-03-20T08:49:16.139 INFO:journalctl@ceph.osd.7.smithi194.stdout:Mar 20 08:49:15 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-osd-7[44872]: 2024-03-20T08:49:15.861+0000 7f37d5961700 -1 osd.7 71 heartbeat_check: no reply from 172.21.15.183:6814 osd.1 since back 2024-03-20T08:48:26.147608+0000 front 2024-03-20T08:48:26.147671+0000 (oldest deadline 2024-03-20T08:48:50.847640+0000) 2024-03-20T08:49:16.139 INFO:journalctl@ceph.osd.7.smithi194.stdout:Mar 20 08:49:15 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-osd-7[44872]: 2024-03-20T08:49:15.861+0000 7f37d5961700 -1 osd.7 71 heartbeat_check: no reply from 172.21.15.183:6822 osd.2 since back 2024-03-20T08:48:36.148515+0000 front 2024-03-20T08:48:36.148487+0000 (oldest deadline 2024-03-20T08:48:59.048455+0000) 2024-03-20T08:49:16.139 INFO:journalctl@ceph.osd.7.smithi194.stdout:Mar 20 08:49:15 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-osd-7[44872]: 2024-03-20T08:49:15.861+0000 7f37d5961700 -1 osd.7 71 heartbeat_check: no reply from 172.21.15.183:6830 osd.3 since back 2024-03-20T08:48:44.949598+0000 front 2024-03-20T08:48:44.949663+0000 (oldest deadline 2024-03-20T08:49:10.849451+0000) 2024-03-20T08:49:16.139 INFO:journalctl@ceph.osd.7.smithi194.stdout:Mar 20 08:49:15 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-osd-7[44872]: 2024-03-20T08:49:15.862+0000 7f37d5961700 -1 osd.7 71 heartbeat_check: no reply from 172.21.15.194:6804 osd.4 since back 2024-03-20T08:48:50.849864+0000 front 2024-03-20T08:48:50.849888+0000 (oldest deadline 2024-03-20T08:49:14.949775+0000) 2024-03-20T08:49:16.139 INFO:journalctl@ceph.osd.7.smithi194.stdout:Mar 20 08:49:15 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-osd-7[44872]: 2024-03-20T08:49:15.862+0000 7f37d5961700 -1 osd.7 71 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.14556.0:149 2.0 2:353ec8f9:::gateway.conf:head [getxattr epoch in=5b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e71) 2024-03-20T08:49:17.139 INFO:journalctl@ceph.osd.7.smithi194.stdout:Mar 20 08:49:16 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-osd-7[44872]: 2024-03-20T08:49:16.828+0000 7f37d5961700 -1 osd.7 71 heartbeat_check: no reply from 172.21.15.183:6806 osd.0 since back 2024-03-20T08:48:16.746885+0000 front 2024-03-20T08:48:16.746994+0000 (oldest deadline 2024-03-20T08:48:42.046867+0000) 2024-03-20T08:49:17.140 INFO:journalctl@ceph.osd.7.smithi194.stdout:Mar 20 08:49:16 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-osd-7[44872]: 2024-03-20T08:49:16.828+0000 7f37d5961700 -1 osd.7 71 heartbeat_check: no reply from 172.21.15.183:6814 osd.1 since back 2024-03-20T08:48:26.147608+0000 front 2024-03-20T08:48:26.147671+0000 (oldest deadline 2024-03-20T08:48:50.847640+0000) 2024-03-20T08:49:17.140 INFO:journalctl@ceph.osd.7.smithi194.stdout:Mar 20 08:49:16 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-osd-7[44872]: 2024-03-20T08:49:16.828+0000 7f37d5961700 -1 osd.7 71 heartbeat_check: no reply from 172.21.15.183:6822 osd.2 since back 2024-03-20T08:48:36.148515+0000 front 2024-03-20T08:48:36.148487+0000 (oldest deadline 2024-03-20T08:48:59.048455+0000) 2024-03-20T08:49:17.140 INFO:journalctl@ceph.osd.7.smithi194.stdout:Mar 20 08:49:16 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-osd-7[44872]: 2024-03-20T08:49:16.828+0000 7f37d5961700 -1 osd.7 71 heartbeat_check: no reply from 172.21.15.183:6830 osd.3 since back 2024-03-20T08:48:44.949598+0000 front 2024-03-20T08:48:44.949663+0000 (oldest deadline 2024-03-20T08:49:10.849451+0000) 2024-03-20T08:49:17.140 INFO:journalctl@ceph.osd.7.smithi194.stdout:Mar 20 08:49:16 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-osd-7[44872]: 2024-03-20T08:49:16.828+0000 7f37d5961700 -1 osd.7 71 heartbeat_check: no reply from 172.21.15.194:6804 osd.4 since back 2024-03-20T08:48:50.849864+0000 front 2024-03-20T08:48:50.849888+0000 (oldest deadline 2024-03-20T08:49:14.949775+0000) 2024-03-20T08:49:17.140 INFO:journalctl@ceph.osd.7.smithi194.stdout:Mar 20 08:49:16 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-osd-7[44872]: 2024-03-20T08:49:16.828+0000 7f37d5961700 -1 osd.7 71 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.14556.0:149 2.0 2:353ec8f9:::gateway.conf:head [getxattr epoch in=5b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e71) 2024-03-20T08:49:18.139 INFO:journalctl@ceph.osd.7.smithi194.stdout:Mar 20 08:49:17 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-osd-7[44872]: 2024-03-20T08:49:17.795+0000 7f37d5961700 -1 osd.7 71 heartbeat_check: no reply from 172.21.15.183:6806 osd.0 since back 2024-03-20T08:48:16.746885+0000 front 2024-03-20T08:48:16.746994+0000 (oldest deadline 2024-03-20T08:48:42.046867+0000) 2024-03-20T08:49:18.139 INFO:journalctl@ceph.osd.7.smithi194.stdout:Mar 20 08:49:17 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-osd-7[44872]: 2024-03-20T08:49:17.795+0000 7f37d5961700 -1 osd.7 71 heartbeat_check: no reply from 172.21.15.183:6814 osd.1 since back 2024-03-20T08:48:26.147608+0000 front 2024-03-20T08:48:26.147671+0000 (oldest deadline 2024-03-20T08:48:50.847640+0000) 2024-03-20T08:49:18.139 INFO:journalctl@ceph.osd.7.smithi194.stdout:Mar 20 08:49:17 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-osd-7[44872]: 2024-03-20T08:49:17.795+0000 7f37d5961700 -1 osd.7 71 heartbeat_check: no reply from 172.21.15.183:6822 osd.2 since back 2024-03-20T08:48:36.148515+0000 front 2024-03-20T08:48:36.148487+0000 (oldest deadline 2024-03-20T08:48:59.048455+0000) 2024-03-20T08:49:18.139 INFO:journalctl@ceph.osd.7.smithi194.stdout:Mar 20 08:49:17 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-osd-7[44872]: 2024-03-20T08:49:17.795+0000 7f37d5961700 -1 osd.7 71 heartbeat_check: no reply from 172.21.15.183:6830 osd.3 since back 2024-03-20T08:48:44.949598+0000 front 2024-03-20T08:48:44.949663+0000 (oldest deadline 2024-03-20T08:49:10.849451+0000) 2024-03-20T08:49:18.139 INFO:journalctl@ceph.osd.7.smithi194.stdout:Mar 20 08:49:17 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-osd-7[44872]: 2024-03-20T08:49:17.795+0000 7f37d5961700 -1 osd.7 71 heartbeat_check: no reply from 172.21.15.194:6804 osd.4 since back 2024-03-20T08:48:50.849864+0000 front 2024-03-20T08:48:50.849888+0000 (oldest deadline 2024-03-20T08:49:14.949775+0000) 2024-03-20T08:49:18.139 INFO:journalctl@ceph.osd.7.smithi194.stdout:Mar 20 08:49:17 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-osd-7[44872]: 2024-03-20T08:49:17.796+0000 7f37d5961700 -1 osd.7 71 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.14556.0:149 2.0 2:353ec8f9:::gateway.conf:head [getxattr epoch in=5b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e71) 2024-03-20T08:49:19.138 INFO:journalctl@ceph.osd.7.smithi194.stdout:Mar 20 08:49:18 smithi194 podman[64094]: 2024-03-20 08:49:18.864543949 +0000 UTC m=+5.651266879 container died 007108f102c240cf2b498ade99e6c26477a89bb34e6672cbfa628e75b6110506 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:976a8b2d06e3c6bfd12488297e8584dd704cc06c3e976c7e0f718c13e238e912, name=ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-osd-7, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.build-date=20240307, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, RELEASE=wip-yuri5-testing-2024-03-18-1144-e7e80a6, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux ) 2024-03-20T08:49:19.633 INFO:journalctl@ceph.osd.7.smithi194.stdout:Mar 20 08:49:19 smithi194 podman[64094]: 2024-03-20 08:49:19.267234385 +0000 UTC m=+6.053957312 container cleanup 007108f102c240cf2b498ade99e6c26477a89bb34e6672cbfa628e75b6110506 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:976a8b2d06e3c6bfd12488297e8584dd704cc06c3e976c7e0f718c13e238e912, name=ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-osd-7, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-yuri5-testing-2024-03-18-1144-e7e80a6, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, ceph=True, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240307, org.label-schema.schema-version=1.0) 2024-03-20T08:49:19.633 INFO:journalctl@ceph.osd.7.smithi194.stdout:Mar 20 08:49:19 smithi194 bash[64094]: ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-osd-7 2024-03-20T08:49:19.633 INFO:journalctl@ceph.osd.7.smithi194.stdout:Mar 20 08:49:19 smithi194 podman[64368]: 2024-03-20 08:49:19.501605172 +0000 UTC m=+0.632115842 container remove 007108f102c240cf2b498ade99e6c26477a89bb34e6672cbfa628e75b6110506 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:976a8b2d06e3c6bfd12488297e8584dd704cc06c3e976c7e0f718c13e238e912, name=ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-osd-7, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_CLEAN=True, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, RELEASE=wip-yuri5-testing-2024-03-18-1144-e7e80a6, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.build-date=20240307) 2024-03-20T08:49:21.297 INFO:journalctl@ceph.osd.7.smithi194.stdout:Mar 20 08:49:21 smithi194 podman[64548]: 2024-03-20 08:49:20.93319718 +0000 UTC m=+0.350732844 image pull 9ceba81fb2c8d8682cb159527017d591637b153b3aa4a390ffa8586db6dcda00 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:976a8b2d06e3c6bfd12488297e8584dd704cc06c3e976c7e0f718c13e238e912 2024-03-20T08:49:21.297 INFO:journalctl@ceph.osd.7.smithi194.stdout:Mar 20 08:49:21 smithi194 podman[64548]: 2024-03-20 08:49:21.048066157 +0000 UTC m=+0.465601820 container create e6f558dff9d07f99da3ab5e598b83dc8894d45ee27f9734f096ba123f4972937 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:976a8b2d06e3c6bfd12488297e8584dd704cc06c3e976c7e0f718c13e238e912, name=ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-osd-7-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, RELEASE=wip-yuri5-testing-2024-03-18-1144-e7e80a6, GIT_BRANCH=HEAD, GIT_CLEAN=True, ceph=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240307) 2024-03-20T08:49:21.297 INFO:journalctl@ceph.osd.7.smithi194.stdout:Mar 20 08:49:21 smithi194 podman[64548]: 2024-03-20 08:49:21.254654769 +0000 UTC m=+0.672190442 container init e6f558dff9d07f99da3ab5e598b83dc8894d45ee27f9734f096ba123f4972937 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:976a8b2d06e3c6bfd12488297e8584dd704cc06c3e976c7e0f718c13e238e912, name=ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-osd-7-deactivate, GIT_CLEAN=True, ceph=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, RELEASE=wip-yuri5-testing-2024-03-18-1144-e7e80a6) 2024-03-20T08:49:21.298 INFO:journalctl@ceph.osd.7.smithi194.stdout:Mar 20 08:49:21 smithi194 podman[64548]: 2024-03-20 08:49:21.258034942 +0000 UTC m=+0.675570606 container start e6f558dff9d07f99da3ab5e598b83dc8894d45ee27f9734f096ba123f4972937 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:976a8b2d06e3c6bfd12488297e8584dd704cc06c3e976c7e0f718c13e238e912, name=ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-osd-7-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.build-date=20240307, ceph=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, RELEASE=wip-yuri5-testing-2024-03-18-1144-e7e80a6, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS) 2024-03-20T08:49:21.578 INFO:journalctl@ceph.osd.7.smithi194.stdout:Mar 20 08:49:21 smithi194 podman[64548]: 2024-03-20 08:49:21.363547425 +0000 UTC m=+0.781083096 container attach e6f558dff9d07f99da3ab5e598b83dc8894d45ee27f9734f096ba123f4972937 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:976a8b2d06e3c6bfd12488297e8584dd704cc06c3e976c7e0f718c13e238e912, name=ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-osd-7-deactivate, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.build-date=20240307, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , RELEASE=wip-yuri5-testing-2024-03-18-1144-e7e80a6) 2024-03-20T08:49:21.579 INFO:journalctl@ceph.prometheus.a.smithi194.stdout:Mar 20 08:49:21 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-prometheus-a[49402]: ts=2024-03-20T08:49:21.295Z caller=refresh.go:99 level=error component="discovery manager scrape" discovery=http config=ceph-exporter msg="Unable to refresh target groups" err="Get \"http://172.21.15.183:8765/sd/prometheus/sd-config?service=ceph-exporter\": dial tcp 172.21.15.183:8765: connect: connection refused" 2024-03-20T08:49:21.579 INFO:journalctl@ceph.prometheus.a.smithi194.stdout:Mar 20 08:49:21 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-prometheus-a[49402]: ts=2024-03-20T08:49:21.295Z caller=refresh.go:99 level=error component="discovery manager scrape" discovery=http config=nvmeof msg="Unable to refresh target groups" err="Get \"http://172.21.15.183:8765/sd/prometheus/sd-config?service=nvmeof\": dial tcp 172.21.15.183:8765: connect: connection refused" 2024-03-20T08:49:21.579 INFO:journalctl@ceph.prometheus.a.smithi194.stdout:Mar 20 08:49:21 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-prometheus-a[49402]: ts=2024-03-20T08:49:21.298Z caller=refresh.go:99 level=error component="discovery manager scrape" discovery=http config=ceph msg="Unable to refresh target groups" err="Get \"http://172.21.15.183:8765/sd/prometheus/sd-config?service=mgr-prometheus\": dial tcp 172.21.15.183:8765: connect: connection refused" 2024-03-20T08:49:21.832 INFO:journalctl@ceph.osd.7.smithi194.stdout:Mar 20 08:49:21 smithi194 podman[64548]: 2024-03-20 08:49:21.6577483 +0000 UTC m=+1.075283966 container died e6f558dff9d07f99da3ab5e598b83dc8894d45ee27f9734f096ba123f4972937 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:976a8b2d06e3c6bfd12488297e8584dd704cc06c3e976c7e0f718c13e238e912, name=ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-osd-7-deactivate, org.label-schema.license=GPLv2, org.label-schema.build-date=20240307, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.vendor=CentOS, RELEASE=wip-yuri5-testing-2024-03-18-1144-e7e80a6, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_CLEAN=True) 2024-03-20T08:49:22.833 DEBUG:teuthology.orchestra.run.smithi194:> sudo pkill -f 'journalctl -f -n 0 -u ceph-4c3f3eba-e695-11ee-95c9-87774f69a715@osd.7.service' 2024-03-20T08:49:22.880 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-20T08:49:22.880 INFO:tasks.cephadm.osd.7:Stopped osd.7 2024-03-20T08:49:22.881 INFO:tasks.cephadm.ceph.rgw.foo.a:Stopping rgw.foo.a... 2024-03-20T08:49:22.881 DEBUG:teuthology.orchestra.run.smithi183:> sudo systemctl stop ceph-4c3f3eba-e695-11ee-95c9-87774f69a715@rgw.foo.a 2024-03-20T08:49:23.204 INFO:journalctl@ceph.rgw.foo.a.smithi183.stdout:Mar 20 08:49:22 smithi183 systemd[1]: Stopping Ceph rgw.foo.a for 4c3f3eba-e695-11ee-95c9-87774f69a715... 2024-03-20T08:49:23.605 INFO:journalctl@ceph.rgw.foo.a.smithi183.stdout:Mar 20 08:49:23 smithi183 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-rgw-foo-a[50972]: 2024-03-20T08:49:23.205+0000 7f353b5bb700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/radosgw -n client.rgw.foo.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-20T08:49:23.605 INFO:journalctl@ceph.rgw.foo.a.smithi183.stdout:Mar 20 08:49:23 smithi183 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-rgw-foo-a[50972]: 2024-03-20T08:49:23.205+0000 7f3549e40ac0 -1 shutting down 2024-03-20T08:49:33.505 INFO:journalctl@ceph.rgw.foo.a.smithi183.stdout:Mar 20 08:49:33 smithi183 bash[75527]: time="2024-03-20T08:49:33Z" level=warning msg="StopSignal SIGTERM failed to stop container ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-rgw-foo-a in 10 seconds, resorting to SIGKILL" 2024-03-20T08:49:33.505 INFO:journalctl@ceph.rgw.foo.a.smithi183.stdout:Mar 20 08:49:33 smithi183 podman[75527]: 2024-03-20 08:49:33.23243976 +0000 UTC m=+10.157946673 container died b6a7f433ee926a5b9cb822a30c765ef8755debd41b6cf58e517d619bdb0b2964 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:976a8b2d06e3c6bfd12488297e8584dd704cc06c3e976c7e0f718c13e238e912, name=ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-rgw-foo-a, io.buildah.version=1.33.5, org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-yuri5-testing-2024-03-18-1144-e7e80a6, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=) 2024-03-20T08:49:33.896 INFO:journalctl@ceph.rgw.foo.a.smithi183.stdout:Mar 20 08:49:33 smithi183 podman[75527]: 2024-03-20 08:49:33.604586742 +0000 UTC m=+10.530093701 container cleanup b6a7f433ee926a5b9cb822a30c765ef8755debd41b6cf58e517d619bdb0b2964 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:976a8b2d06e3c6bfd12488297e8584dd704cc06c3e976c7e0f718c13e238e912, name=ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-rgw-foo-a, org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , ceph=True, RELEASE=wip-yuri5-testing-2024-03-18-1144-e7e80a6, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-20T08:49:33.897 INFO:journalctl@ceph.rgw.foo.a.smithi183.stdout:Mar 20 08:49:33 smithi183 bash[75527]: ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-rgw-foo-a 2024-03-20T08:49:34.013 DEBUG:teuthology.orchestra.run.smithi183:> sudo pkill -f 'journalctl -f -n 0 -u ceph-4c3f3eba-e695-11ee-95c9-87774f69a715@rgw.foo.a.service' 2024-03-20T08:49:34.051 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-20T08:49:34.051 INFO:tasks.cephadm.ceph.rgw.foo.a:Stopped rgw.foo.a 2024-03-20T08:49:34.052 INFO:tasks.cephadm.prometheus.a:Stopping prometheus.a... 2024-03-20T08:49:34.052 DEBUG:teuthology.orchestra.run.smithi194:> sudo systemctl stop ceph-4c3f3eba-e695-11ee-95c9-87774f69a715@prometheus.a 2024-03-20T08:49:34.389 INFO:journalctl@ceph.prometheus.a.smithi194.stdout:Mar 20 08:49:34 smithi194 systemd[1]: Stopping Ceph prometheus.a for 4c3f3eba-e695-11ee-95c9-87774f69a715... 2024-03-20T08:49:34.666 INFO:journalctl@ceph.prometheus.a.smithi194.stdout:Mar 20 08:49:34 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-prometheus-a[49402]: ts=2024-03-20T08:49:34.391Z caller=main.go:840 level=warn msg="Received SIGTERM, exiting gracefully..." 2024-03-20T08:49:34.666 INFO:journalctl@ceph.prometheus.a.smithi194.stdout:Mar 20 08:49:34 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-prometheus-a[49402]: ts=2024-03-20T08:49:34.391Z caller=main.go:864 level=info msg="Stopping scrape discovery manager..." 2024-03-20T08:49:34.666 INFO:journalctl@ceph.prometheus.a.smithi194.stdout:Mar 20 08:49:34 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-prometheus-a[49402]: ts=2024-03-20T08:49:34.391Z caller=main.go:878 level=info msg="Stopping notify discovery manager..." 2024-03-20T08:49:34.666 INFO:journalctl@ceph.prometheus.a.smithi194.stdout:Mar 20 08:49:34 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-prometheus-a[49402]: ts=2024-03-20T08:49:34.391Z caller=manager.go:988 level=info component="rule manager" msg="Stopping rule manager..." 2024-03-20T08:49:34.666 INFO:journalctl@ceph.prometheus.a.smithi194.stdout:Mar 20 08:49:34 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-prometheus-a[49402]: ts=2024-03-20T08:49:34.391Z caller=main.go:860 level=info msg="Scrape discovery manager stopped" 2024-03-20T08:49:34.666 INFO:journalctl@ceph.prometheus.a.smithi194.stdout:Mar 20 08:49:34 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-prometheus-a[49402]: ts=2024-03-20T08:49:34.391Z caller=main.go:874 level=info msg="Notify discovery manager stopped" 2024-03-20T08:49:34.667 INFO:journalctl@ceph.prometheus.a.smithi194.stdout:Mar 20 08:49:34 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-prometheus-a[49402]: ts=2024-03-20T08:49:34.391Z caller=manager.go:998 level=info component="rule manager" msg="Rule manager stopped" 2024-03-20T08:49:34.667 INFO:journalctl@ceph.prometheus.a.smithi194.stdout:Mar 20 08:49:34 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-prometheus-a[49402]: ts=2024-03-20T08:49:34.391Z caller=main.go:915 level=info msg="Stopping scrape manager..." 2024-03-20T08:49:34.667 INFO:journalctl@ceph.prometheus.a.smithi194.stdout:Mar 20 08:49:34 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-prometheus-a[49402]: ts=2024-03-20T08:49:34.391Z caller=main.go:907 level=info msg="Scrape manager stopped" 2024-03-20T08:49:34.667 INFO:journalctl@ceph.prometheus.a.smithi194.stdout:Mar 20 08:49:34 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-prometheus-a[49402]: ts=2024-03-20T08:49:34.391Z caller=notifier.go:606 level=info component=notifier msg="Stopping notification manager..." 2024-03-20T08:49:34.667 INFO:journalctl@ceph.prometheus.a.smithi194.stdout:Mar 20 08:49:34 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-prometheus-a[49402]: ts=2024-03-20T08:49:34.391Z caller=main.go:1135 level=info msg="Notifier manager stopped" 2024-03-20T08:49:34.667 INFO:journalctl@ceph.prometheus.a.smithi194.stdout:Mar 20 08:49:34 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-prometheus-a[49402]: ts=2024-03-20T08:49:34.391Z caller=main.go:1147 level=info msg="See you next time!" 2024-03-20T08:49:34.667 INFO:journalctl@ceph.prometheus.a.smithi194.stdout:Mar 20 08:49:34 smithi194 podman[65107]: 2024-03-20 08:49:34.401560248 +0000 UTC m=+0.157861764 container died 3ad955ff33fcffd28c741d406a5929a664909fcc4007e6ab164e8c38026b1228 (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-20T08:49:35.108 INFO:journalctl@ceph.prometheus.a.smithi194.stdout:Mar 20 08:49:34 smithi194 podman[65107]: 2024-03-20 08:49:34.8658296 +0000 UTC m=+0.622131124 container cleanup 3ad955ff33fcffd28c741d406a5929a664909fcc4007e6ab164e8c38026b1228 (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-20T08:49:35.108 INFO:journalctl@ceph.prometheus.a.smithi194.stdout:Mar 20 08:49:34 smithi194 bash[65107]: ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-prometheus-a 2024-03-20T08:49:35.270 DEBUG:teuthology.orchestra.run.smithi194:> sudo pkill -f 'journalctl -f -n 0 -u ceph-4c3f3eba-e695-11ee-95c9-87774f69a715@prometheus.a.service' 2024-03-20T08:49:35.307 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-20T08:49:35.308 INFO:tasks.cephadm.prometheus.a:Stopped prometheus.a 2024-03-20T08:49:35.308 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 4c3f3eba-e695-11ee-95c9-87774f69a715 --force --keep-logs 2024-03-20T08:49:35.499 INFO:teuthology.orchestra.run.smithi183.stdout:Deleting cluster with fsid: 4c3f3eba-e695-11ee-95c9-87774f69a715 2024-03-20T08:49:37.643 INFO:journalctl@ceph.alertmanager.a.smithi183.stdout:Mar 20 08:49:37 smithi183 systemd[1]: Stopping Ceph alertmanager.a for 4c3f3eba-e695-11ee-95c9-87774f69a715... 2024-03-20T08:49:37.909 INFO:journalctl@ceph.alertmanager.a.smithi183.stdout:Mar 20 08:49:37 smithi183 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-alertmanager-a[71868]: ts=2024-03-20T08:49:37.642Z caller=main.go:583 level=info msg="Received SIGTERM, exiting gracefully..." 2024-03-20T08:49:37.909 INFO:journalctl@ceph.alertmanager.a.smithi183.stdout:Mar 20 08:49:37 smithi183 podman[76063]: 2024-03-20 08:49:37.663165023 +0000 UTC m=+0.122318069 container died 3aa83d988989e0cf6b2a0d2389fcf7b0e8343aad495911a18e5ca7cedfafe732 (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-03-20T08:49:38.204 INFO:journalctl@ceph.alertmanager.a.smithi183.stdout:Mar 20 08:49:38 smithi183 podman[76063]: 2024-03-20 08:49:38.133990429 +0000 UTC m=+0.593143477 container cleanup 3aa83d988989e0cf6b2a0d2389fcf7b0e8343aad495911a18e5ca7cedfafe732 (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-03-20T08:49:38.204 INFO:journalctl@ceph.alertmanager.a.smithi183.stdout:Mar 20 08:49:38 smithi183 bash[76063]: ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-alertmanager-a 2024-03-20T08:49:38.577 INFO:journalctl@ceph.alertmanager.a.smithi183.stdout:Mar 20 08:49:38 smithi183 podman[76077]: 2024-03-20 08:49:38.309954917 +0000 UTC m=+0.644049371 container remove 3aa83d988989e0cf6b2a0d2389fcf7b0e8343aad495911a18e5ca7cedfafe732 (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-03-20T08:49:38.577 INFO:journalctl@ceph.alertmanager.a.smithi183.stdout:Mar 20 08:49:38 smithi183 podman[76077]: 2024-03-20 08:49:38.393933966 +0000 UTC m=+0.728028378 volume remove 7c9d4b409337d1a7b07b7ae6c0c0c1b109abae5d7589001e0603cfd35b01e746 2024-03-20T08:49:38.954 INFO:journalctl@ceph.node-exporter.a.smithi183.stdout:Mar 20 08:49:38 smithi183 systemd[1]: Stopping Ceph node-exporter.a for 4c3f3eba-e695-11ee-95c9-87774f69a715... 2024-03-20T08:49:38.954 INFO:journalctl@ceph.alertmanager.a.smithi183.stdout:Mar 20 08:49:38 smithi183 systemd[1]: ceph-4c3f3eba-e695-11ee-95c9-87774f69a715@alertmanager.a.service: Deactivated successfully. 2024-03-20T08:49:38.954 INFO:journalctl@ceph.alertmanager.a.smithi183.stdout:Mar 20 08:49:38 smithi183 systemd[1]: Stopped Ceph alertmanager.a for 4c3f3eba-e695-11ee-95c9-87774f69a715. 2024-03-20T08:49:39.313 INFO:journalctl@ceph.node-exporter.a.smithi183.stdout:Mar 20 08:49:39 smithi183 podman[76189]: 2024-03-20 08:49:39.056852328 +0000 UTC m=+0.111807903 container died 0dace97e4fc4562217de3d754934c883b3949f55529b51ef7f64abc4d49e3d5b (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-node-exporter-a, maintainer=The Prometheus Authors ) 2024-03-20T08:49:39.679 INFO:journalctl@ceph.node-exporter.a.smithi183.stdout:Mar 20 08:49:39 smithi183 podman[76189]: 2024-03-20 08:49:39.404439392 +0000 UTC m=+0.459394965 container cleanup 0dace97e4fc4562217de3d754934c883b3949f55529b51ef7f64abc4d49e3d5b (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-node-exporter-a, maintainer=The Prometheus Authors ) 2024-03-20T08:49:39.679 INFO:journalctl@ceph.node-exporter.a.smithi183.stdout:Mar 20 08:49:39 smithi183 bash[76189]: ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-node-exporter-a 2024-03-20T08:49:39.954 INFO:journalctl@ceph.node-exporter.a.smithi183.stdout:Mar 20 08:49:39 smithi183 podman[76202]: 2024-03-20 08:49:39.680046068 +0000 UTC m=+0.623100618 container remove 0dace97e4fc4562217de3d754934c883b3949f55529b51ef7f64abc4d49e3d5b (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-node-exporter-a, maintainer=The Prometheus Authors ) 2024-03-20T08:49:39.955 INFO:journalctl@ceph.node-exporter.a.smithi183.stdout:Mar 20 08:49:39 smithi183 systemd[1]: ceph-4c3f3eba-e695-11ee-95c9-87774f69a715@node-exporter.a.service: Main process exited, code=exited, status=143/n/a 2024-03-20T08:49:39.955 INFO:journalctl@ceph.node-exporter.a.smithi183.stdout:Mar 20 08:49:39 smithi183 systemd[1]: ceph-4c3f3eba-e695-11ee-95c9-87774f69a715@node-exporter.a.service: Failed with result 'exit-code'. 2024-03-20T08:49:39.955 INFO:journalctl@ceph.node-exporter.a.smithi183.stdout:Mar 20 08:49:39 smithi183 systemd[1]: Stopped Ceph node-exporter.a for 4c3f3eba-e695-11ee-95c9-87774f69a715. 2024-03-20T08:49:39.955 INFO:journalctl@ceph.node-exporter.a.smithi183.stdout:Mar 20 08:49:39 smithi183 systemd[1]: ceph-4c3f3eba-e695-11ee-95c9-87774f69a715@node-exporter.a.service: Consumed 1.804s CPU time. 2024-03-20T08:49:40.485 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 4c3f3eba-e695-11ee-95c9-87774f69a715 --force --keep-logs 2024-03-20T08:49:42.579 INFO:teuthology.orchestra.run.smithi194.stdout:Deleting cluster with fsid: 4c3f3eba-e695-11ee-95c9-87774f69a715 2024-03-20T08:49:44.389 INFO:journalctl@ceph.iscsi.iscsi.a.smithi194.stdout:Mar 20 08:49:44 smithi194 systemd[1]: Stopping Ceph iscsi.iscsi.a for 4c3f3eba-e695-11ee-95c9-87774f69a715... 2024-03-20T08:49:44.389 INFO:journalctl@ceph.iscsi.iscsi.a.smithi194.stdout:Mar 20 08:49:44 smithi194 ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-iscsi-iscsi-a[47467]: debug Shutdown received 2024-03-20T08:49:54.607 INFO:journalctl@ceph.iscsi.iscsi.a.smithi194.stdout:Mar 20 08:49:54 smithi194 bash[65958]: time="2024-03-20T08:49:54Z" level=warning msg="StopSignal SIGTERM failed to stop container ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-iscsi-iscsi-a in 10 seconds, resorting to SIGKILL" 2024-03-20T08:49:54.607 INFO:journalctl@ceph.iscsi.iscsi.a.smithi194.stdout:Mar 20 08:49:54 smithi194 podman[65958]: 2024-03-20 08:49:54.362015032 +0000 UTC m=+10.116429151 container died 589372ff43adb43d2f6756641db21093909ce04f47b08e8e7acd6a4f3b5a206f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:976a8b2d06e3c6bfd12488297e8584dd704cc06c3e976c7e0f718c13e238e912, name=ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-iscsi-iscsi-a, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, ceph=True, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307, RELEASE=wip-yuri5-testing-2024-03-18-1144-e7e80a6, GIT_CLEAN=True, org.label-schema.vendor=CentOS) 2024-03-20T08:49:54.888 INFO:journalctl@ceph.iscsi.iscsi.a.smithi194.stdout:Mar 20 08:49:54 smithi194 podman[65958]: 2024-03-20 08:49:54.756433435 +0000 UTC m=+10.510847546 container cleanup 589372ff43adb43d2f6756641db21093909ce04f47b08e8e7acd6a4f3b5a206f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:976a8b2d06e3c6bfd12488297e8584dd704cc06c3e976c7e0f718c13e238e912, name=ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-iscsi-iscsi-a, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.build-date=20240307, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-yuri5-testing-2024-03-18-1144-e7e80a6, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-20T08:49:54.889 INFO:journalctl@ceph.iscsi.iscsi.a.smithi194.stdout:Mar 20 08:49:54 smithi194 bash[65958]: ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-iscsi-iscsi-a 2024-03-20T08:49:55.308 INFO:journalctl@ceph.iscsi.iscsi.a.smithi194.stdout:Mar 20 08:49:54 smithi194 podman[65973]: 2024-03-20 08:49:54.974144681 +0000 UTC m=+0.608717468 container remove 589372ff43adb43d2f6756641db21093909ce04f47b08e8e7acd6a4f3b5a206f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:976a8b2d06e3c6bfd12488297e8584dd704cc06c3e976c7e0f718c13e238e912, name=ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-iscsi-iscsi-a, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, RELEASE=wip-yuri5-testing-2024-03-18-1144-e7e80a6, org.label-schema.vendor=CentOS, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240307, GIT_BRANCH=HEAD, ceph=True) 2024-03-20T08:49:55.309 INFO:journalctl@ceph.iscsi.iscsi.a.smithi194.stdout:Mar 20 08:49:54 smithi194 systemd[1]: ceph-4c3f3eba-e695-11ee-95c9-87774f69a715@iscsi.iscsi.a.service: Main process exited, code=exited, status=137/n/a 2024-03-20T08:49:55.309 INFO:journalctl@ceph.iscsi.iscsi.a.smithi194.stdout:Mar 20 08:49:55 smithi194 systemd[1]: ceph-4c3f3eba-e695-11ee-95c9-87774f69a715@iscsi.iscsi.a.service: Failed with result 'exit-code'. 2024-03-20T08:49:55.309 INFO:journalctl@ceph.iscsi.iscsi.a.smithi194.stdout:Mar 20 08:49:55 smithi194 systemd[1]: Stopped Ceph iscsi.iscsi.a for 4c3f3eba-e695-11ee-95c9-87774f69a715. 2024-03-20T08:49:55.309 INFO:journalctl@ceph.iscsi.iscsi.a.smithi194.stdout:Mar 20 08:49:55 smithi194 systemd[1]: ceph-4c3f3eba-e695-11ee-95c9-87774f69a715@iscsi.iscsi.a.service: Consumed 2.904s CPU time. 2024-03-20T08:49:56.883 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Mar 20 08:49:56 smithi194 systemd[1]: Stopping Ceph grafana.a for 4c3f3eba-e695-11ee-95c9-87774f69a715... 2024-03-20T08:49:57.139 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Mar 20 08:49:56 smithi194 podman[66233]: 2024-03-20 08:49:56.883470894 +0000 UTC m=+0.122789271 container died 98413bced1ff941af42ed1cd3fa177f8ed0baf311366b04e93983b1087f7462d (image=quay.io/ceph/ceph-grafana:9.4.12, name=ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-grafana-a, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-239.1651231664, description=Ceph Grafana Container, io.openshift.tags=base rhel8, name=ubi8, vcs-type=git, com.redhat.build-host=cpt-1007.osbs.prod.upshift.rdu2.redhat.com, io.buildah.version=1.32.0, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, summary=Grafana Container configured for Ceph mgr/dashboard integration, vendor=Red Hat, Inc., com.redhat.component=ubi8-container, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., architecture=x86_64, build-date=2022-04-29T11:48:11.055897, distribution-scope=public, version=8.5, maintainer=Paul Cuzner , io.k8s.display-name=Red Hat Universal Base Image 8, io.openshift.expose-services=, release=239.1651231664, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI) 2024-03-20T08:49:57.639 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Mar 20 08:49:57 smithi194 podman[66233]: 2024-03-20 08:49:57.297530359 +0000 UTC m=+0.536848734 container cleanup 98413bced1ff941af42ed1cd3fa177f8ed0baf311366b04e93983b1087f7462d (image=quay.io/ceph/ceph-grafana:9.4.12, name=ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-grafana-a, name=ubi8, io.buildah.version=1.32.0, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, maintainer=Paul Cuzner , release=239.1651231664, build-date=2022-04-29T11:48:11.055897, version=8.5, summary=Grafana Container configured for Ceph mgr/dashboard integration, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, com.redhat.component=ubi8-container, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., architecture=x86_64, io.k8s.display-name=Red Hat Universal Base Image 8, vcs-type=git, vendor=Red Hat, Inc., com.redhat.build-host=cpt-1007.osbs.prod.upshift.rdu2.redhat.com, distribution-scope=public, description=Ceph Grafana Container, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-239.1651231664, io.openshift.expose-services=, io.openshift.tags=base rhel8) 2024-03-20T08:49:57.639 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Mar 20 08:49:57 smithi194 bash[66233]: ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-grafana-a 2024-03-20T08:49:57.639 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Mar 20 08:49:57 smithi194 podman[66247]: 2024-03-20 08:49:57.490395257 +0000 UTC m=+0.607605261 container remove 98413bced1ff941af42ed1cd3fa177f8ed0baf311366b04e93983b1087f7462d (image=quay.io/ceph/ceph-grafana:9.4.12, name=ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-grafana-a, io.buildah.version=1.32.0, maintainer=Paul Cuzner , com.redhat.build-host=cpt-1007.osbs.prod.upshift.rdu2.redhat.com, vendor=Red Hat, Inc., description=Ceph Grafana Container, distribution-scope=public, name=ubi8, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-239.1651231664, architecture=x86_64, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.openshift.tags=base rhel8, com.redhat.component=ubi8-container, version=8.5, io.openshift.expose-services=, release=239.1651231664, summary=Grafana Container configured for Ceph mgr/dashboard integration, build-date=2022-04-29T11:48:11.055897, vcs-type=git, io.k8s.display-name=Red Hat Universal Base Image 8) 2024-03-20T08:49:57.639 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Mar 20 08:49:57 smithi194 systemd[1]: ceph-4c3f3eba-e695-11ee-95c9-87774f69a715@grafana.a.service: Main process exited, code=exited, status=143/n/a 2024-03-20T08:49:58.139 INFO:journalctl@ceph.node-exporter.b.smithi194.stdout:Mar 20 08:49:57 smithi194 systemd[1]: Stopping Ceph node-exporter.b for 4c3f3eba-e695-11ee-95c9-87774f69a715... 2024-03-20T08:49:58.139 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Mar 20 08:49:57 smithi194 systemd[1]: ceph-4c3f3eba-e695-11ee-95c9-87774f69a715@grafana.a.service: Failed with result 'exit-code'. 2024-03-20T08:49:58.139 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Mar 20 08:49:57 smithi194 systemd[1]: Stopped Ceph grafana.a for 4c3f3eba-e695-11ee-95c9-87774f69a715. 2024-03-20T08:49:58.139 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Mar 20 08:49:57 smithi194 systemd[1]: ceph-4c3f3eba-e695-11ee-95c9-87774f69a715@grafana.a.service: Consumed 4.575s CPU time. 2024-03-20T08:49:58.518 INFO:journalctl@ceph.node-exporter.b.smithi194.stdout:Mar 20 08:49:58 smithi194 podman[66360]: 2024-03-20 08:49:58.178755056 +0000 UTC m=+0.114056548 container died fed6ffe6922c5f9411b6b1993e01d015efccce0cb8bb290aa1c7240dc831eea6 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-node-exporter-b, maintainer=The Prometheus Authors ) 2024-03-20T08:49:58.518 INFO:journalctl@ceph.node-exporter.b.smithi194.stdout:Mar 20 08:49:58 smithi194 podman[66360]: 2024-03-20 08:49:58.51822254 +0000 UTC m=+0.453524033 container cleanup fed6ffe6922c5f9411b6b1993e01d015efccce0cb8bb290aa1c7240dc831eea6 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-node-exporter-b, maintainer=The Prometheus Authors ) 2024-03-20T08:49:58.802 INFO:journalctl@ceph.node-exporter.b.smithi194.stdout:Mar 20 08:49:58 smithi194 bash[66360]: ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-node-exporter-b 2024-03-20T08:49:59.108 INFO:journalctl@ceph.node-exporter.b.smithi194.stdout:Mar 20 08:49:58 smithi194 podman[66374]: 2024-03-20 08:49:58.802264342 +0000 UTC m=+0.628355772 container remove fed6ffe6922c5f9411b6b1993e01d015efccce0cb8bb290aa1c7240dc831eea6 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-4c3f3eba-e695-11ee-95c9-87774f69a715-node-exporter-b, maintainer=The Prometheus Authors ) 2024-03-20T08:49:59.109 INFO:journalctl@ceph.node-exporter.b.smithi194.stdout:Mar 20 08:49:58 smithi194 systemd[1]: ceph-4c3f3eba-e695-11ee-95c9-87774f69a715@node-exporter.b.service: Main process exited, code=exited, status=143/n/a 2024-03-20T08:49:59.109 INFO:journalctl@ceph.node-exporter.b.smithi194.stdout:Mar 20 08:49:58 smithi194 systemd[1]: ceph-4c3f3eba-e695-11ee-95c9-87774f69a715@node-exporter.b.service: Failed with result 'exit-code'. 2024-03-20T08:49:59.109 INFO:journalctl@ceph.node-exporter.b.smithi194.stdout:Mar 20 08:49:58 smithi194 systemd[1]: Stopped Ceph node-exporter.b for 4c3f3eba-e695-11ee-95c9-87774f69a715. 2024-03-20T08:49:59.109 INFO:journalctl@ceph.node-exporter.b.smithi194.stdout:Mar 20 08:49:58 smithi194 systemd[1]: ceph-4c3f3eba-e695-11ee-95c9-87774f69a715@node-exporter.b.service: Consumed 1.688s CPU time. 2024-03-20T08:49:59.553 DEBUG:teuthology.orchestra.run.smithi183:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-03-20T08:49:59.588 DEBUG:teuthology.orchestra.run.smithi194:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-03-20T08:49:59.618 INFO:tasks.cephadm:Archiving crash dumps... 2024-03-20T08:49:59.620 DEBUG:teuthology.misc:Transferring archived files from smithi183:/var/lib/ceph/4c3f3eba-e695-11ee-95c9-87774f69a715/crash to /home/teuthworker/archive/yuriw-2024-03-19_00:09:45-rados-wip-yuri5-testing-2024-03-18-1144-distro-default-smithi/7610111/remote/smithi183/crash 2024-03-20T08:49:59.620 DEBUG:teuthology.orchestra.run.smithi183:> sudo tar c -f - -C /var/lib/ceph/4c3f3eba-e695-11ee-95c9-87774f69a715/crash -- . 2024-03-20T08:49:59.668 INFO:teuthology.orchestra.run.smithi183.stderr:tar: /var/lib/ceph/4c3f3eba-e695-11ee-95c9-87774f69a715/crash: Cannot open: No such file or directory 2024-03-20T08:49:59.668 INFO:teuthology.orchestra.run.smithi183.stderr:tar: Error is not recoverable: exiting now 2024-03-20T08:49:59.670 DEBUG:teuthology.misc:Transferring archived files from smithi194:/var/lib/ceph/4c3f3eba-e695-11ee-95c9-87774f69a715/crash to /home/teuthworker/archive/yuriw-2024-03-19_00:09:45-rados-wip-yuri5-testing-2024-03-18-1144-distro-default-smithi/7610111/remote/smithi194/crash 2024-03-20T08:49:59.670 DEBUG:teuthology.orchestra.run.smithi194:> sudo tar c -f - -C /var/lib/ceph/4c3f3eba-e695-11ee-95c9-87774f69a715/crash -- . 2024-03-20T08:49:59.727 INFO:teuthology.orchestra.run.smithi194.stderr:tar: /var/lib/ceph/4c3f3eba-e695-11ee-95c9-87774f69a715/crash: Cannot open: No such file or directory 2024-03-20T08:49:59.728 INFO:teuthology.orchestra.run.smithi194.stderr:tar: Error is not recoverable: exiting now 2024-03-20T08:49:59.729 INFO:tasks.cephadm:Checking cluster log for badness... 2024-03-20T08:49:59.729 DEBUG:teuthology.orchestra.run.smithi183:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/4c3f3eba-e695-11ee-95c9-87774f69a715/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v MON_DOWN | head -n 1 2024-03-20T08:49:59.758 INFO:teuthology.orchestra.run.smithi183.stdout:2024-03-20T08:41:51.585112+0000 mon.a (mon.0) 240 : cluster [WRN] Health check failed: 1 stray daemon(s) not managed by cephadm (CEPHADM_STRAY_DAEMON) 2024-03-20T08:49:59.761 WARNING:tasks.cephadm:Found errors (ERR|WRN|SEC) in cluster log 2024-03-20T08:49:59.762 DEBUG:teuthology.orchestra.run.smithi183:> sudo egrep '\[SEC\]' /var/log/ceph/4c3f3eba-e695-11ee-95c9-87774f69a715/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v MON_DOWN | head -n 1 2024-03-20T08:49:59.827 DEBUG:teuthology.orchestra.run.smithi183:> sudo egrep '\[ERR\]' /var/log/ceph/4c3f3eba-e695-11ee-95c9-87774f69a715/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v MON_DOWN | head -n 1 2024-03-20T08:49:59.897 DEBUG:teuthology.orchestra.run.smithi183:> sudo egrep '\[WRN\]' /var/log/ceph/4c3f3eba-e695-11ee-95c9-87774f69a715/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v MON_DOWN | head -n 1 2024-03-20T08:49:59.966 INFO:teuthology.orchestra.run.smithi183.stdout:2024-03-20T08:41:51.585112+0000 mon.a (mon.0) 240 : cluster [WRN] Health check failed: 1 stray daemon(s) not managed by cephadm (CEPHADM_STRAY_DAEMON) 2024-03-20T08:49:59.967 INFO:tasks.cephadm:Compressing logs... 2024-03-20T08:49:59.967 DEBUG:teuthology.orchestra.run.smithi183:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-03-20T08:50:00.009 DEBUG:teuthology.orchestra.run.smithi194:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-03-20T08:50:00.034 INFO:teuthology.orchestra.run.smithi183.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-03-20T08:50:00.034 INFO:teuthology.orchestra.run.smithi183.stderr:: No such file or directory 2024-03-20T08:50:00.034 INFO:teuthology.orchestra.run.smithi183.stderr:gzip -5 --verbose -- /var/log/ceph/4c3f3eba-e695-11ee-95c9-87774f69a715/ceph-mon.a.log 2024-03-20T08:50:00.035 INFO:teuthology.orchestra.run.smithi183.stderr:gzip -5 --verbose -- /var/log/ceph/4c3f3eba-e695-11ee-95c9-87774f69a715/ceph.audit.log 2024-03-20T08:50:00.035 INFO:teuthology.orchestra.run.smithi183.stderr:/var/log/ceph/cephadm.log: /var/log/ceph/4c3f3eba-e695-11ee-95c9-87774f69a715/ceph-mon.a.log: gzip -5 --verbose -- /var/log/ceph/4c3f3eba-e695-11ee-95c9-87774f69a715/ceph-client.rgw.foo.a.log 2024-03-20T08:50:00.035 INFO:teuthology.orchestra.run.smithi183.stderr:/var/log/ceph/4c3f3eba-e695-11ee-95c9-87774f69a715/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/4c3f3eba-e695-11ee-95c9-87774f69a715/ceph-osd.1.log 2024-03-20T08:50:00.036 INFO:teuthology.orchestra.run.smithi183.stderr:gzip -5 --verbose -- /var/log/ceph/4c3f3eba-e695-11ee-95c9-87774f69a715/ceph-osd.3.log 2024-03-20T08:50:00.040 INFO:teuthology.orchestra.run.smithi183.stderr:/var/log/ceph/4c3f3eba-e695-11ee-95c9-87774f69a715/ceph-client.rgw.foo.a.log: 57.8%/var/log/ceph/4c3f3eba-e695-11ee-95c9-87774f69a715/ceph-osd.1.log: -- replaced with /var/log/ceph/4c3f3eba-e695-11ee-95c9-87774f69a715/ceph-client.rgw.foo.a.log.gzgzip -5 --verbose -- /var/log/ceph/4c3f3eba-e695-11ee-95c9-87774f69a715/ceph-mgr.y.log 2024-03-20T08:50:00.041 INFO:teuthology.orchestra.run.smithi183.stderr: 2024-03-20T08:50:00.041 INFO:teuthology.orchestra.run.smithi183.stderr:/var/log/ceph/4c3f3eba-e695-11ee-95c9-87774f69a715/ceph-osd.3.log: gzip -5 --verbose -- /var/log/ceph/4c3f3eba-e695-11ee-95c9-87774f69a715/ceph-osd.0.log 2024-03-20T08:50:00.041 INFO:teuthology.orchestra.run.smithi183.stderr:/var/log/ceph/4c3f3eba-e695-11ee-95c9-87774f69a715/ceph-mgr.y.log: /var/log/ceph/4c3f3eba-e695-11ee-95c9-87774f69a715/ceph-osd.0.log: gzip -5 --verbose -- /var/log/ceph/4c3f3eba-e695-11ee-95c9-87774f69a715/ceph.cephadm.log 2024-03-20T08:50:00.041 INFO:teuthology.orchestra.run.smithi183.stderr: 94.0% -- replaced with /var/log/ceph/4c3f3eba-e695-11ee-95c9-87774f69a715/ceph.audit.log.gz 2024-03-20T08:50:00.041 INFO:teuthology.orchestra.run.smithi183.stderr: 86.8% -- replaced with /var/log/ceph/cephadm.log.gz 2024-03-20T08:50:00.041 INFO:teuthology.orchestra.run.smithi183.stderr:gzip -5 --verbose -- /var/log/ceph/4c3f3eba-e695-11ee-95c9-87774f69a715/ceph.log 2024-03-20T08:50:00.041 INFO:teuthology.orchestra.run.smithi183.stderr:gzip/var/log/ceph/4c3f3eba-e695-11ee-95c9-87774f69a715/ceph.cephadm.log: -5 --verbose -- /var/log/ceph/4c3f3eba-e695-11ee-95c9-87774f69a715/ceph-osd.2.log 2024-03-20T08:50:00.041 INFO:teuthology.orchestra.run.smithi183.stderr:/var/log/ceph/4c3f3eba-e695-11ee-95c9-87774f69a715/ceph.log: 89.1% -- replaced with /var/log/ceph/4c3f3eba-e695-11ee-95c9-87774f69a715/ceph.cephadm.log.gz 2024-03-20T08:50:00.044 INFO:teuthology.orchestra.run.smithi183.stderr:/var/log/ceph/4c3f3eba-e695-11ee-95c9-87774f69a715/ceph-osd.2.log: 91.9% -- replaced with /var/log/ceph/4c3f3eba-e695-11ee-95c9-87774f69a715/ceph.log.gz 2024-03-20T08:50:00.044 INFO:teuthology.orchestra.run.smithi183.stderr:gzip -5 --verbose -- /var/log/ceph/4c3f3eba-e695-11ee-95c9-87774f69a715/ceph-volume.log 2024-03-20T08:50:00.045 INFO:teuthology.orchestra.run.smithi183.stderr:gzip -5 --verbose -- /var/log/ceph/4c3f3eba-e695-11ee-95c9-87774f69a715/ceph-mon.c.log 2024-03-20T08:50:00.045 INFO:teuthology.orchestra.run.smithi194.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-03-20T08:50:00.046 INFO:teuthology.orchestra.run.smithi194.stderr:: No such file or directory 2024-03-20T08:50:00.046 INFO:teuthology.orchestra.run.smithi194.stderr:gzip -5 --verbose -- /var/log/ceph/4c3f3eba-e695-11ee-95c9-87774f69a715/ceph-osd.6.log 2024-03-20T08:50:00.046 INFO:teuthology.orchestra.run.smithi194.stderr:gzip -5 --verbose --/var/log/ceph/cephadm.log: /var/log/ceph/4c3f3eba-e695-11ee-95c9-87774f69a715/ceph.audit.log 2024-03-20T08:50:00.046 INFO:teuthology.orchestra.run.smithi194.stderr:gzip -5 --verbose -- /var/log/ceph/4c3f3eba-e695-11ee-95c9-87774f69a715/ceph-osd.4.log 2024-03-20T08:50:00.046 INFO:teuthology.orchestra.run.smithi194.stderr:gzip -5 --verbose -- /var/log/ceph/4c3f3eba-e695-11ee-95c9-87774f69a715/ceph-osd.7.log 2024-03-20T08:50:00.046 INFO:teuthology.orchestra.run.smithi194.stderr:/var/log/ceph/4c3f3eba-e695-11ee-95c9-87774f69a715/ceph-osd.6.log: /var/log/ceph/4c3f3eba-e695-11ee-95c9-87774f69a715/ceph.audit.log: gzip/var/log/ceph/4c3f3eba-e695-11ee-95c9-87774f69a715/ceph-osd.4.log: -5 --verbose -- /var/log/ceph/4c3f3eba-e695-11ee-95c9-87774f69a715/ceph-osd.5.log 2024-03-20T08:50:00.046 INFO:teuthology.orchestra.run.smithi194.stderr:/var/log/ceph/4c3f3eba-e695-11ee-95c9-87774f69a715/ceph-osd.7.log: gzip -5 --verbose -- /var/log/ceph/4c3f3eba-e695-11ee-95c9-87774f69a715/ceph-mgr.x.log 2024-03-20T08:50:00.046 INFO:teuthology.orchestra.run.smithi194.stderr:/var/log/ceph/4c3f3eba-e695-11ee-95c9-87774f69a715/ceph-osd.5.log: gzip -5 --verbose -- /var/log/ceph/4c3f3eba-e695-11ee-95c9-87774f69a715/tcmu-runner.log 2024-03-20T08:50:00.046 INFO:teuthology.orchestra.run.smithi194.stderr:/var/log/ceph/4c3f3eba-e695-11ee-95c9-87774f69a715/ceph-mgr.x.log: gzip -5 --verbose -- /var/log/ceph/4c3f3eba-e695-11ee-95c9-87774f69a715/ceph.cephadm.log 2024-03-20T08:50:00.046 INFO:teuthology.orchestra.run.smithi194.stderr:/var/log/ceph/4c3f3eba-e695-11ee-95c9-87774f69a715/tcmu-runner.log: 67.1% -- replaced with /var/log/ceph/4c3f3eba-e695-11ee-95c9-87774f69a715/tcmu-runner.log.gz 2024-03-20T08:50:00.046 INFO:teuthology.orchestra.run.smithi194.stderr: 90.4% -- replaced with /var/log/ceph/4c3f3eba-e695-11ee-95c9-87774f69a715/ceph.audit.log.gz 2024-03-20T08:50:00.046 INFO:teuthology.orchestra.run.smithi194.stderr: 83.5% -- replaced with /var/log/ceph/cephadm.log.gz 2024-03-20T08:50:00.046 INFO:teuthology.orchestra.run.smithi194.stderr:gzip -5 --verbose --/var/log/ceph/4c3f3eba-e695-11ee-95c9-87774f69a715/ceph.cephadm.log: /var/log/ceph/4c3f3eba-e695-11ee-95c9-87774f69a715/ceph.log 2024-03-20T08:50:00.046 INFO:teuthology.orchestra.run.smithi194.stderr:gzip -5 --verbose -- /var/log/ceph/4c3f3eba-e695-11ee-95c9-87774f69a715/ceph-volume.log 2024-03-20T08:50:00.046 INFO:teuthology.orchestra.run.smithi194.stderr: 81.7% -- replaced with /var/log/ceph/4c3f3eba-e695-11ee-95c9-87774f69a715/ceph.cephadm.log.gz 2024-03-20T08:50:00.047 INFO:teuthology.orchestra.run.smithi194.stderr:/var/log/ceph/4c3f3eba-e695-11ee-95c9-87774f69a715/ceph.log: gzip -5 --verbose -- /var/log/ceph/4c3f3eba-e695-11ee-95c9-87774f69a715/ceph-mon.b.log 2024-03-20T08:50:00.047 INFO:teuthology.orchestra.run.smithi194.stderr:/var/log/ceph/4c3f3eba-e695-11ee-95c9-87774f69a715/ceph-volume.log: 86.1% -- replaced with /var/log/ceph/4c3f3eba-e695-11ee-95c9-87774f69a715/ceph.log.gz 2024-03-20T08:50:00.047 INFO:teuthology.orchestra.run.smithi194.stderr: 90.6% -- replaced with /var/log/ceph/4c3f3eba-e695-11ee-95c9-87774f69a715/ceph-mgr.x.log.gz 2024-03-20T08:50:00.101 INFO:teuthology.orchestra.run.smithi194.stderr:/var/log/ceph/4c3f3eba-e695-11ee-95c9-87774f69a715/ceph-mon.b.log: 90.1% -- replaced with /var/log/ceph/4c3f3eba-e695-11ee-95c9-87774f69a715/ceph-volume.log.gz 2024-03-20T08:50:00.102 INFO:teuthology.orchestra.run.smithi183.stderr:/var/log/ceph/4c3f3eba-e695-11ee-95c9-87774f69a715/ceph-volume.log: /var/log/ceph/4c3f3eba-e695-11ee-95c9-87774f69a715/ceph-mon.c.log: 90.1% -- replaced with /var/log/ceph/4c3f3eba-e695-11ee-95c9-87774f69a715/ceph-volume.log.gz 2024-03-20T08:50:00.171 INFO:teuthology.orchestra.run.smithi194.stderr: 92.2% -- replaced with /var/log/ceph/4c3f3eba-e695-11ee-95c9-87774f69a715/ceph-mon.b.log.gz 2024-03-20T08:50:00.200 INFO:teuthology.orchestra.run.smithi183.stderr: 88.1% -- replaced with /var/log/ceph/4c3f3eba-e695-11ee-95c9-87774f69a715/ceph-mgr.y.log.gz 2024-03-20T08:50:00.231 INFO:teuthology.orchestra.run.smithi194.stderr: 93.2% -- replaced with /var/log/ceph/4c3f3eba-e695-11ee-95c9-87774f69a715/ceph-osd.7.log.gz 2024-03-20T08:50:00.250 INFO:teuthology.orchestra.run.smithi183.stderr: 92.3% -- replaced with /var/log/ceph/4c3f3eba-e695-11ee-95c9-87774f69a715/ceph-mon.c.log.gz 2024-03-20T08:50:00.255 INFO:teuthology.orchestra.run.smithi194.stderr: 93.0% -- replaced with /var/log/ceph/4c3f3eba-e695-11ee-95c9-87774f69a715/ceph-osd.6.log.gz 2024-03-20T08:50:00.265 INFO:teuthology.orchestra.run.smithi194.stderr: 93.1% -- replaced with /var/log/ceph/4c3f3eba-e695-11ee-95c9-87774f69a715/ceph-osd.5.log.gz 2024-03-20T08:50:00.275 INFO:teuthology.orchestra.run.smithi194.stderr: 93.4% -- replaced with /var/log/ceph/4c3f3eba-e695-11ee-95c9-87774f69a715/ceph-osd.4.log.gz 2024-03-20T08:50:00.277 INFO:teuthology.orchestra.run.smithi194.stderr: 2024-03-20T08:50:00.277 INFO:teuthology.orchestra.run.smithi194.stderr:real 0m0.253s 2024-03-20T08:50:00.277 INFO:teuthology.orchestra.run.smithi194.stderr:user 0m1.040s 2024-03-20T08:50:00.277 INFO:teuthology.orchestra.run.smithi194.stderr:sys 0m0.054s 2024-03-20T08:50:00.287 INFO:teuthology.orchestra.run.smithi183.stderr: 93.0% -- replaced with /var/log/ceph/4c3f3eba-e695-11ee-95c9-87774f69a715/ceph-osd.2.log.gz 2024-03-20T08:50:00.320 INFO:teuthology.orchestra.run.smithi183.stderr: 93.4% -- replaced with /var/log/ceph/4c3f3eba-e695-11ee-95c9-87774f69a715/ceph-osd.0.log.gz 2024-03-20T08:50:00.351 INFO:teuthology.orchestra.run.smithi183.stderr: 93.4% -- replaced with /var/log/ceph/4c3f3eba-e695-11ee-95c9-87774f69a715/ceph-osd.3.log.gz 2024-03-20T08:50:00.390 INFO:teuthology.orchestra.run.smithi183.stderr: 93.5% -- replaced with /var/log/ceph/4c3f3eba-e695-11ee-95c9-87774f69a715/ceph-osd.1.log.gz 2024-03-20T08:50:00.418 INFO:teuthology.orchestra.run.smithi183.stderr: 91.3% -- replaced with /var/log/ceph/4c3f3eba-e695-11ee-95c9-87774f69a715/ceph-mon.a.log.gz 2024-03-20T08:50:00.420 INFO:teuthology.orchestra.run.smithi183.stderr: 2024-03-20T08:50:00.420 INFO:teuthology.orchestra.run.smithi183.stderr:real 0m0.400s 2024-03-20T08:50:00.420 INFO:teuthology.orchestra.run.smithi183.stderr:user 0m1.877s 2024-03-20T08:50:00.420 INFO:teuthology.orchestra.run.smithi183.stderr:sys 0m0.088s 2024-03-20T08:50:00.421 INFO:tasks.cephadm:Archiving logs... 2024-03-20T08:50:00.421 DEBUG:teuthology.misc:Transferring archived files from smithi183:/var/log/ceph to /home/teuthworker/archive/yuriw-2024-03-19_00:09:45-rados-wip-yuri5-testing-2024-03-18-1144-distro-default-smithi/7610111/remote/smithi183/log 2024-03-20T08:50:00.422 DEBUG:teuthology.orchestra.run.smithi183:> sudo tar c -f - -C /var/log/ceph -- . 2024-03-20T08:50:00.720 DEBUG:teuthology.misc:Transferring archived files from smithi194:/var/log/ceph to /home/teuthworker/archive/yuriw-2024-03-19_00:09:45-rados-wip-yuri5-testing-2024-03-18-1144-distro-default-smithi/7610111/remote/smithi194/log 2024-03-20T08:50:00.721 DEBUG:teuthology.orchestra.run.smithi194:> sudo tar c -f - -C /var/log/ceph -- . 2024-03-20T08:50:00.898 INFO:tasks.cephadm:Removing cluster... 2024-03-20T08:50:00.898 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 4c3f3eba-e695-11ee-95c9-87774f69a715 --force 2024-03-20T08:50:01.081 INFO:teuthology.orchestra.run.smithi183.stdout:Deleting cluster with fsid: 4c3f3eba-e695-11ee-95c9-87774f69a715 2024-03-20T08:50:01.198 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 4c3f3eba-e695-11ee-95c9-87774f69a715 --force 2024-03-20T08:50:01.380 INFO:teuthology.orchestra.run.smithi194.stdout:Deleting cluster with fsid: 4c3f3eba-e695-11ee-95c9-87774f69a715 2024-03-20T08:50:01.504 INFO:tasks.cephadm:Removing cephadm ... 2024-03-20T08:50:01.505 DEBUG:teuthology.orchestra.run.smithi183:> rm -rf /home/ubuntu/cephtest/cephadm 2024-03-20T08:50:01.520 DEBUG:teuthology.orchestra.run.smithi194:> rm -rf /home/ubuntu/cephtest/cephadm 2024-03-20T08:50:01.536 INFO:tasks.cephadm:Teardown complete 2024-03-20T08:50:01.536 DEBUG:teuthology.run_tasks:Unwinding manager nvme_loop 2024-03-20T08:50:01.548 INFO:tasks.nvme_loop:Disconnecting nvme_loop smithi183:/dev/vg_nvme/lv_1... 2024-03-20T08:50:01.549 DEBUG:teuthology.orchestra.run.smithi183:> sudo nvme disconnect -n lv_1 2024-03-20T08:50:01.699 INFO:teuthology.orchestra.run.smithi183.stdout:NQN:lv_1 disconnected 1 controller(s) 2024-03-20T08:50:01.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-20T08:50:01.701 INFO:tasks.nvme_loop:Disconnecting nvme_loop smithi183:/dev/vg_nvme/lv_2... 2024-03-20T08:50:01.701 DEBUG:teuthology.orchestra.run.smithi183:> sudo nvme disconnect -n lv_2 2024-03-20T08:50:01.845 INFO:teuthology.orchestra.run.smithi183.stdout:NQN:lv_2 disconnected 1 controller(s) 2024-03-20T08:50:01.846 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-20T08:50:01.847 INFO:tasks.nvme_loop:Disconnecting nvme_loop smithi183:/dev/vg_nvme/lv_3... 2024-03-20T08:50:01.847 DEBUG:teuthology.orchestra.run.smithi183:> sudo nvme disconnect -n lv_3 2024-03-20T08:50:01.982 INFO:teuthology.orchestra.run.smithi183.stdout:NQN:lv_3 disconnected 1 controller(s) 2024-03-20T08:50:01.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-20T08:50:01.985 INFO:tasks.nvme_loop:Disconnecting nvme_loop smithi183:/dev/vg_nvme/lv_4... 2024-03-20T08:50:01.985 DEBUG:teuthology.orchestra.run.smithi183:> sudo nvme disconnect -n lv_4 2024-03-20T08:50:02.126 INFO:teuthology.orchestra.run.smithi183.stdout:NQN:lv_4 disconnected 1 controller(s) 2024-03-20T08:50:02.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-20T08:50:02.128 DEBUG:teuthology.orchestra.run.smithi183:> set -ex 2024-03-20T08:50:02.129 DEBUG:teuthology.orchestra.run.smithi183:> sudo dd of=/scratch_devs 2024-03-20T08:50:02.156 INFO:tasks.nvme_loop:Disconnecting nvme_loop smithi194:/dev/vg_nvme/lv_1... 2024-03-20T08:50:02.156 DEBUG:teuthology.orchestra.run.smithi194:> sudo nvme disconnect -n lv_1 2024-03-20T08:50:02.294 INFO:teuthology.orchestra.run.smithi194.stdout:NQN:lv_1 disconnected 1 controller(s) 2024-03-20T08:50:02.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-20T08:50:02.296 INFO:tasks.nvme_loop:Disconnecting nvme_loop smithi194:/dev/vg_nvme/lv_2... 2024-03-20T08:50:02.297 DEBUG:teuthology.orchestra.run.smithi194:> sudo nvme disconnect -n lv_2 2024-03-20T08:50:02.443 INFO:teuthology.orchestra.run.smithi194.stdout:NQN:lv_2 disconnected 1 controller(s) 2024-03-20T08:50:02.508 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-20T08:50:02.509 INFO:tasks.nvme_loop:Disconnecting nvme_loop smithi194:/dev/vg_nvme/lv_3... 2024-03-20T08:50:02.509 DEBUG:teuthology.orchestra.run.smithi194:> sudo nvme disconnect -n lv_3 2024-03-20T08:50:02.649 INFO:teuthology.orchestra.run.smithi194.stdout:NQN:lv_3 disconnected 1 controller(s) 2024-03-20T08:50:02.650 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-20T08:50:02.651 INFO:tasks.nvme_loop:Disconnecting nvme_loop smithi194:/dev/vg_nvme/lv_4... 2024-03-20T08:50:02.652 DEBUG:teuthology.orchestra.run.smithi194:> sudo nvme disconnect -n lv_4 2024-03-20T08:50:02.807 INFO:teuthology.orchestra.run.smithi194.stdout:NQN:lv_4 disconnected 1 controller(s) 2024-03-20T08:50:02.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-20T08:50:02.809 DEBUG:teuthology.orchestra.run.smithi194:> set -ex 2024-03-20T08:50:02.809 DEBUG:teuthology.orchestra.run.smithi194:> sudo dd of=/scratch_devs 2024-03-20T08:50:02.837 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-03-20T08:50:02.846 INFO:teuthology.task.clock:Checking final clock skew... 2024-03-20T08:50:02.846 DEBUG:teuthology.orchestra.run.smithi183:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-20T08:50:02.849 DEBUG:teuthology.orchestra.run.smithi194:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-20T08:50:02.863 INFO:teuthology.orchestra.run.smithi183.stderr:bash: line 1: ntpq: command not found 2024-03-20T08:50:02.868 INFO:teuthology.orchestra.run.smithi183.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-20T08:50:02.868 INFO:teuthology.orchestra.run.smithi183.stdout:=============================================================================== 2024-03-20T08:50:02.868 INFO:teuthology.orchestra.run.smithi183.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 12 -1131us[-1131us] +/- 76ms 2024-03-20T08:50:02.868 INFO:teuthology.orchestra.run.smithi183.stdout:^* hv02.front.sepia.ceph.com 3 6 377 13 -127us[ -122us] +/- 49ms 2024-03-20T08:50:02.868 INFO:teuthology.orchestra.run.smithi183.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 14 +925us[ +930us] +/- 75ms 2024-03-20T08:50:02.868 INFO:teuthology.orchestra.run.smithi183.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2024-03-20T08:50:02.892 INFO:teuthology.orchestra.run.smithi194.stderr:bash: line 1: ntpq: command not found 2024-03-20T08:50:02.896 INFO:teuthology.orchestra.run.smithi194.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-20T08:50:02.896 INFO:teuthology.orchestra.run.smithi194.stdout:=============================================================================== 2024-03-20T08:50:02.896 INFO:teuthology.orchestra.run.smithi194.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 14 -1019us[-1143us] +/- 76ms 2024-03-20T08:50:02.896 INFO:teuthology.orchestra.run.smithi194.stdout:^* hv02.front.sepia.ceph.com 3 6 377 12 -7865ns[ -133us] +/- 49ms 2024-03-20T08:50:02.896 INFO:teuthology.orchestra.run.smithi194.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 12 +1036us[+1036us] +/- 75ms 2024-03-20T08:50:02.896 INFO:teuthology.orchestra.run.smithi194.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2024-03-20T08:50:02.897 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-03-20T08:50:02.907 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-03-20T08:50:02.908 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-03-20T08:50:02.926 DEBUG:teuthology.orchestra.run.smithi183:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-03-20T08:50:03.006 DEBUG:teuthology.orchestra.run.smithi194:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-03-20T08:50:03.078 DEBUG:teuthology.orchestra.run.smithi183:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 2024-03-20T08:50:03.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-20T08:50:03.110 DEBUG:teuthology.orchestra.run.smithi194:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 2024-03-20T08:50:03.139 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-20T08:50:03.140 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-03-20T08:50:03.150 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-03-20T08:50:03.168 INFO:teuthology.task.internal:Duration was 930.922733 seconds 2024-03-20T08:50:03.169 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-03-20T08:50:03.202 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-03-20T08:50:03.202 DEBUG:teuthology.orchestra.run.smithi183:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-03-20T08:50:03.205 DEBUG:teuthology.orchestra.run.smithi194:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-03-20T08:50:03.248 INFO:teuthology.orchestra.run.smithi183.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-20T08:50:03.249 INFO:teuthology.orchestra.run.smithi194.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-20T08:50:03.743 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-03-20T08:50:03.743 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi183.front.sepia.ceph.com 2024-03-20T08:50:03.744 DEBUG:teuthology.orchestra.run.smithi183:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-03-20T08:50:03.764 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi194.front.sepia.ceph.com 2024-03-20T08:50:03.764 DEBUG:teuthology.orchestra.run.smithi194:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-03-20T08:50:03.793 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-03-20T08:50:03.793 DEBUG:teuthology.orchestra.run.smithi183:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-03-20T08:50:03.806 DEBUG:teuthology.orchestra.run.smithi194:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-03-20T08:50:03.861 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-03-20T08:50:03.862 DEBUG:teuthology.orchestra.run.smithi183:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-03-20T08:50:03.876 DEBUG:teuthology.orchestra.run.smithi194:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-03-20T08:50:04.132 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-03-20T08:50:04.142 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-03-20T08:50:04.143 DEBUG:teuthology.orchestra.run.smithi183:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-03-20T08:50:04.169 DEBUG:teuthology.orchestra.run.smithi194:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-03-20T08:50:04.196 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-03-20T08:50:04.205 DEBUG:teuthology.orchestra.run.smithi183:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-03-20T08:50:04.211 DEBUG:teuthology.orchestra.run.smithi194:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-03-20T08:50:04.234 INFO:teuthology.orchestra.run.smithi183.stdout:kernel.core_pattern = core 2024-03-20T08:50:04.261 INFO:teuthology.orchestra.run.smithi194.stdout:kernel.core_pattern = core 2024-03-20T08:50:04.290 DEBUG:teuthology.orchestra.run.smithi183:> test -e /home/ubuntu/cephtest/archive/coredump 2024-03-20T08:50:04.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-20T08:50:04.315 DEBUG:teuthology.orchestra.run.smithi194:> test -e /home/ubuntu/cephtest/archive/coredump 2024-03-20T08:50:04.344 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-20T08:50:04.344 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-03-20T08:50:04.355 INFO:teuthology.task.internal:Transferring archived files... 2024-03-20T08:50:04.356 DEBUG:teuthology.misc:Transferring archived files from smithi183:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/yuriw-2024-03-19_00:09:45-rados-wip-yuri5-testing-2024-03-18-1144-distro-default-smithi/7610111/remote/smithi183 2024-03-20T08:50:04.356 DEBUG:teuthology.orchestra.run.smithi183:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-03-20T08:50:04.398 DEBUG:teuthology.misc:Transferring archived files from smithi194:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/yuriw-2024-03-19_00:09:45-rados-wip-yuri5-testing-2024-03-18-1144-distro-default-smithi/7610111/remote/smithi194 2024-03-20T08:50:04.398 DEBUG:teuthology.orchestra.run.smithi194:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-03-20T08:50:04.437 INFO:teuthology.task.internal:Removing archive directory... 2024-03-20T08:50:04.437 DEBUG:teuthology.orchestra.run.smithi183:> rm -rf -- /home/ubuntu/cephtest/archive 2024-03-20T08:50:04.440 DEBUG:teuthology.orchestra.run.smithi194:> rm -rf -- /home/ubuntu/cephtest/archive 2024-03-20T08:50:04.486 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-03-20T08:50:04.496 INFO:teuthology.task.internal:Not uploading archives. 2024-03-20T08:50:04.497 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-03-20T08:50:04.522 INFO:teuthology.task.internal:Tidying up after the test... 2024-03-20T08:50:04.522 DEBUG:teuthology.orchestra.run.smithi183:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-03-20T08:50:04.524 DEBUG:teuthology.orchestra.run.smithi194:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-03-20T08:50:04.536 INFO:teuthology.orchestra.run.smithi183.stdout: 262166 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 20 08:50 /home/ubuntu/cephtest 2024-03-20T08:50:04.540 INFO:teuthology.orchestra.run.smithi194.stdout: 265305 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 20 08:50 /home/ubuntu/cephtest 2024-03-20T08:50:04.541 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-03-20T08:50:04.554 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-03-20T08:50:04.620 INFO:teuthology.nuke:Checking targets against current locks 2024-03-20T08:50:04.646 DEBUG:teuthology.nuke:shortname: smithi183 2024-03-20T08:50:04.646 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-20T08:50:04.672 DEBUG:teuthology.nuke:shortname: smithi194 2024-03-20T08:50:04.672 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-20T08:50:04.675 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi183.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2024-03-19_00:09:45-rados-wip-yuri5-testing-2024-03-18-1144-distro-default-smithi/7610111', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-20 08:26:00.972556', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBE4gCTVxUfs527UFnLA/Kcux6hTHlOG8K4vYYK26DaPAdBmVfF4qKIk0niQNGp2Rmx7qYSVs5eD1tsF0kOzshpo='} 2024-03-20T08:50:04.697 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi194.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2024-03-19_00:09:45-rados-wip-yuri5-testing-2024-03-18-1144-distro-default-smithi/7610111', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-20 08:26:00.974208', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBLUGBjFRNsFwW1fGbSBzj8dKikSjLYtQonwzNpc8xquXi0ye2tjK6uCZ1gkUH4xO573o7ji0knEGNdy4ndlSL8g='} 2024-03-20T08:50:04.727 INFO:teuthology.orchestra.console.smithi183:Power off 2024-03-20T08:50:04.728 DEBUG:teuthology.orchestra.console.smithi183:pexpect command: ipmitool -H smithi183.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-20T08:50:04.752 DEBUG:teuthology.orchestra.console.smithi183:power off output: Chassis Power Control: Down/Off 2024-03-20T08:50:04.753 DEBUG:teuthology.orchestra.console.smithi183:pexpect command: ipmitool -H smithi183.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-20T08:50:04.770 INFO:teuthology.orchestra.console.smithi194:Power off 2024-03-20T08:50:04.771 DEBUG:teuthology.orchestra.console.smithi194:pexpect command: ipmitool -H smithi194.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-20T08:50:04.782 DEBUG:teuthology.orchestra.console.smithi183:check power output: Chassis Power is on 2024-03-20T08:50:04.797 DEBUG:teuthology.orchestra.console.smithi194:power off output: Chassis Power Control: Down/Off 2024-03-20T08:50:04.797 DEBUG:teuthology.orchestra.console.smithi194:pexpect command: ipmitool -H smithi194.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-20T08:50:04.824 DEBUG:teuthology.orchestra.console.smithi194:check power output: Chassis Power is on 2024-03-20T08:50:08.783 DEBUG:teuthology.orchestra.console.smithi183:pexpect command: ipmitool -H smithi183.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-20T08:50:08.825 DEBUG:teuthology.orchestra.console.smithi194:pexpect command: ipmitool -H smithi194.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-20T08:50:08.894 DEBUG:teuthology.orchestra.console.smithi183:check power output: Chassis Power is on 2024-03-20T08:50:08.936 DEBUG:teuthology.orchestra.console.smithi194:check power output: Chassis Power is on 2024-03-20T08:50:12.898 DEBUG:teuthology.orchestra.console.smithi183:pexpect command: ipmitool -H smithi183.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-20T08:50:12.937 DEBUG:teuthology.orchestra.console.smithi194:pexpect command: ipmitool -H smithi194.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-20T08:50:13.012 DEBUG:teuthology.orchestra.console.smithi183:check power output: Chassis Power is off 2024-03-20T08:50:13.047 DEBUG:teuthology.orchestra.console.smithi194:check power output: Chassis Power is off 2024-03-20T08:50:13.113 INFO:teuthology.orchestra.console.smithi183:Power off completed 2024-03-20T08:50:13.149 INFO:teuthology.orchestra.console.smithi194:Power off completed 2024-03-20T08:50:13.253 INFO:teuthology.run:Summary data: description: rados/cephadm/smoke/{0-distro/centos_9.stream 0-nvme-loop agent/on fixed-2 mon_election/connectivity start} duration: 930.922732591629 failure_reason: '"2024-03-20T08:41:51.585112+0000 mon.a (mon.0) 240 : cluster [WRN] Health check failed: 1 stray daemon(s) not managed by cephadm (CEPHADM_STRAY_DAEMON)" in cluster log' owner: scheduled_yuriw@teuthology success: false 2024-03-20T08:50:13.254 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-03-20T08:50:13.333 INFO:teuthology.run:FAIL